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Executive  Summary 

Background 


The  Ministry  of  Treasury  Board  of  the  Government  of  Alberta  (GoA)  approached 
CCAF-FCVI,  Inc.  (CCAF)  to  undertake  a direct  dialogue  with  the  users  of  the  government’s 
Public  Performance  Reports  (PPRs). 

Recognized  by  CCAF  and  others  as  a leader  in  the  field  of 
public  performance  reporting,  the  GoA  wanted  to  build  on 
the  existing  successes  of  its  public  performance  reporting 
regime.  The  purpose  of  this  dialogue  was  to  identify  ways 
the  government  could  improve  its  PPRs  to  better  meet  the 
needs  of  users.  This  was  one  of  the  first  direct  consultations 
of  user  groups  of  its  kind  in  Canada. 

The  GoA,  via  a Working  Group  and  an  Advisory  Committee, 
led  and  managed  all  aspects  of  the  consultation;  CCAF 
facilitated  the  consultations  and  interviews  and  prepared  this  report. 

CCAF  is  a non-profit  foundation  whose  mission  is  to  provide  exemplary  thought  leadership 
and  to  build  both  knowledge  and  capacity  for  effective  governance  and  meaningful 
accountability,  management  and  audit.  This  Alberta  consultation  project  forms  part  of  a 
larger,  multi-jurisdictional  research  project  that  CCAF  is  leading  with  funding  from  the  Alfred 
P.  Sloan  Foundation.  The  larger  project  aims  to  improve  the  quality  and  usage  of  public 
performance  reporting  and  will  combine  key  findings  from  the  Alberta  project  with  results  on 
recent  developments  in  public  performance  reporting  from  other  organizations  and 
governments. 

CCAF  and  Bluesky  Strategy  Group,  Inc.  led  a series  of  consultations  in  Edmonton  with  the 
media,  business  NGOs  (non-governmental  organizations)/professional  associations  and 
social/advocacy  NGOs  (including  a separate  social/advocacy  NGO  group  in  Calgary) 
between  October  and  November  2007.  During  each  of  the  two-hour  consultations, 
participants  engaged  in  a dialogue  and  responded  to  a written  questionnaire. 

Consultations  with  government  and  opposition  Members  of  the  Legislative  Assembly  of 
Alberta  (MLAs)  also  took  place  in  Edmonton,  during  the  week  of  December  10,  2007.  Each 
consultation  lasted  between  45  and  90  minutes.  During  that  week,  there  were  separate 
individual  interviews  with  cabinet  ministers,  a senior  government  official,  members  of  the 
Provincial  Audit  Committee,  and  members  of  the  project’s  Advisory  Committee. 

The  participating  organizations  and  the  questions  posed  to  the  groups  are  listed  in  the 
appendices  to  this  report.  Participants  in  the  consultations  were  also  asked  to  complete  a 
written  survey.  The  survey  had  a 100%  response  rate,  with  31  surveys  completed,  as 
individual  interviewees  were  not  asked  to  complete  the  written  survey.  The  survey 
contained  nine  sections  examining  knowledge,  use,  format,  and  confidence  in  PPRs,  as 
well  as  a section  for  additional  comments/suggestions.  The  Consultation  Survey,  and 
findings  (Appendix  II  and  III)  are  referenced  throughout  this  report. 


Public  performance  reports,  for  the  purpose  of  this 
study,  include  the  Government  of  Alberta  business 
plan,  ministry  business  plans,  the  Government  of 
Alberta’s  annual  report  Measuring  Up,  and 
ministry  annual  reports.  The  financial  statements 
in  these  documents  were  excluded  from  review. 




IMPROVING  PUBLIC  PERFORMANCE  REPORTS 


Findings  and  Recommendations 


The  participants’  ideas  for  enhancing  public  performance  reporting  can  be  summarized 
under  four  principal  themes,  which  form  the  framework  of  this  report.  Flowing  from  these 
themes,  the  consultations  yielded  several  key  findings  - that  is,  the  most  important 
messages  delivered  by  users  and  an  overview  of  CCAF  recommendations  related  to  each 
of  the  key  findings. 

1 . Enhancing  the  Credibility  of  PPRs  - The  credibility  of  performance  reports  was 
one  of  the  most  important  factors  in  the  degree  to  which  most  participants  used  or 
found  value  in  the  reports. 

Participants  suggested  that  credibility  suffers  if  government  appears  to  claim  credit  for 
results  beyond  its  control;  if  government  is  unable  to  admit  failure  (and  to  signal  that  it 
can  learn  from  failures);  if  performance  measures  are  not  consistent  over  time;  or  if  the 

wording  of  reports  obscures  meaning. 

They  asked  for  a more  self-evident 
connection  between  goals  (what  the 
government  wants  to  achieve),  strategies 
(how  it  plans  to  achieve  the  goals),  and 
performance  measures  (how  it  will 
determine  results).  They  wanted  goals, 
strategies  and  performance  measures  to 
be  presented  in  a consistent  manner 
across  ministries  and  from  year-to-year, 
with  explanations  where  there  have  been 
changes  over  time. 

Furthermore,  participants  said  the 
inclusion  of  more  financial  data,  linked  to 
strategies  and  performance  measures, 
would  aid  credibility  by  making  it  easier  to 
determine  if  the  government  is  obtaining 
value  for  money  and  is  adequately  funding 
key  initiatives. 


KEY  FINDINGS 

Credibility 

PPRs  are  more  credible  if  they  emphasize  results  and  outcomes,  and  do  not 
focus  on  the  processes  used  to  achieve  the  results.  If  targets  are  not  met,  PPRs 
should  acknowledge  this,  and  show  what  has  been  learned  from  the  experience. 
Performance  measures  should  be  presented  in  a consistent  manner  over  time  so 
that  performance  results  are  evident;  if  changes  are  made,  they  should  be 
explained.  PPR  language  should  be  clear  and  simple  so  readers  do  not  get  the 
impression  that  the  government  is  attempting  to  hide  something. 

Overview  of  CCAF  recommendations  related  to  each  of  the  key  findings: 

♦ The  Government  of  Alberta  should  assume  a leadership  role  in  the 
development  of  performance  measures  for  horizontal  initiatives  (i.e.  initiatives 
in  which  partners  from  two  or  more  organizations  agree  to  work  towards  the 
achievement  of  shared  outcomes). 

♦ The  Ministry  of  Treasury  Board,  working  with  Executive  Council,  and 
ministries  should  partner  to  better  define  their  strategic  goals  and  determine 
how  their  strategies,  programs  and  policies  contribute  to  achieving  those 
goals.  The  Ministry  of  Treasury  Board  should  also  advise  on  performance 
measures. 

♦ The  Government  of  Alberta  should  encourage  greater  consistency  for  year- 
to-year  performance  measures,  indicate  any  changes  to  the  measures  used 
and  link  financial  information,  such  as  budgets,  more  closely  to  non-financial 
information,  such  as  strategies,  performance  measures  and  targets. 
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2. 


Increasing  the 
Relevance  of  PPRs  - 


KEY  FINDINGS 

Relevance 


Relevant  public 
performance  reports 
help  users  to 
appreciate,  on  a timely 
basis,  what  is  planned, 
what  has  happened 
and  why  it  happened. 

PPRs  are  more 
relevant  to  users  if 


Users  want  to  "follow  the  money"  through  closer  links  between  annual  reports,  business  plans  and 
relevant  financial  reports. 

Different  users  have  different  needs  for  PPRs.  They  may  use  them  for  general  information,  to 
identify  priorities,  for  their  own  strategic  planning  and  policy  development,  to  assess  government's 
performance,  and/or  to  identify  service  and  policy  gaps.  Some  require  much  more  detailed 
information  to  meet  their  specific  needs. 

Overview  of  CCAF  recommendations  related  to  each  of  the  key  findings: 

4 Ministries  should  make  more  detailed  technical  information  available  through  website  links,  and 
provide  contact  names  for  additional  information  to  meet  the  different  needs  of  different  user 
groups. 

4 The  Government  of  Alberta  should  ensure  closer  linkages  between  business  plans,  annual 
reports  and  financial  information  to  offer  the  largest  potential  benefit  across  user  groups. 


users  can  clearly  see 

how  the  strategies  employed  relate  to  the  goals  sought.  Relevance  is  also  increased  if 
performance  information  is  placed  in  context,  i.e.,  if  an  explanation  is  given  for  the 
selection  of  a particular  target. 
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Many  participants  wanted  to  see  a closer  link  between  annual  reports  and  business 
plans.  Media  participants  suggested  that  PPRs  should  state:  a)  “here  is  what  we  said 
we  were  going  to  do”  and  b)  “here  is  what  we  did.” 


3.  Improving  the  Clarity  and  Accessibility  of  PPRs  - To  improve  the  clarity  and 
accessibility  of  PPRs,  many  participants  proposed  that  government  make  greater  use 
of  the  internet  and  of  existing  information  technologies. 

Most  users  said  they  prefer  to  access  PPRs 
through  the  internet.  They  also  expressed  a 
preference  for  a report  card/aggregated  indices 
format  and  a layered  approach  to  reporting 
information.  They  thought  such  an  approach 
would  help  to  meet  different  user  needs. 

NGO  participants  in  particular  want  access  to 
more  of  the  detailed  data  the  GoA  collects  in 
the  course  of  delivering  programs  and 
services.  They  believed  this  could  be  done  via 
hyperlinks  in  the  online  versions  of  PPRs.  This 
would  enable  users  to  “draw  their  own 
conclusions”  on  government  performance  and 
more  easily  combine  government-sourced  data 
with  other  information  sources.  Currently,  the 
users  find  the  data  in  PPRs  generally 
interesting  but  not  particularly  useful  for 
business  purposes. 
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KEY  FINDINGS 

Clarity  and  Accessibility 

Many  participants  believe  that  greater  use  of  information  technology, 
combined  with  robust  management  systems,  could  enable  more  data 
to  be  presented,  without  interpretation,  at  different  layers  of  detail. 

This  would  also  make  it  easier  to  combine  public  performance 
reporting  data  with  other  data. 

Over  time,  making  more  program-level  statistics  (including 
disaggregated  data)  publicly  available  would  lessen  the  demand  on 
PPRs  as  the  primary  source  for  this  type  of  program  management 
information. 

Overview  of  CCAF  recommendations  related  to  each  of  the  key 
findings: 

4 Ministries  should  make  greater  use  of  hyperlinks  in  online 
versions  of  PPRs  so  users  can  "drill  down"  for  more  detailed 
information  on  goals,  strategies  and  performance  measures, 
including  contextual  and  methodological  information.  The  resulting 
"layered  reports"  would  better  meet  diverse  user  needs. 

4 The  Government  of  Alberta  should  review  how  other  jurisdictions 
are  using  new  technologies  to  enhance  their  PPRs  with  an  eye  to 
determining  which  technologies  might  be  cost  effective  in  Alberta. 

V ) 


A decision  on  whether  to  make  program-level  statistics  and  information  more  available 
would  have  implications  beyond  those  considered  in  this  report.  Nevertheless, 
because  such  increased  information  availability  would  have  an  impact  on  how  users 
perceive  PPRs,  this  report  includes  participants’  comments  on  this  issue. 
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4.  Engaging  Users  in  PPRs  - The  Government  of  Alberta  often  consults  citizens  on 
policy  issues.  However,  such  consultations  do  not  usually  include  an  explicit 
discussion  of  performance  measures,  targets,  and  reporting  mechanisms.  This  current 
low  level  of  engagement  regarding  these  matters  results  in  a low  level  of  familiarity 
with  ministry  goals,  strategies  and  performance  measures. 

The  consultations  revealed  that  user  groups  generally  feel  disengaged  from  the  public 
performance  reporting  process.  NGOs  made  it  clear  that  their  perception  of  the 
credibility,  relevance  and  usefulness  of  PPRs  is  directly  influenced  by  the  extent  to 
which  they  are  engaged  in  the  formulation  of  PPRs  and  in  any  dialogue  that  may  occur 
after  the  publication  of  PPRs.  Legislators  said  they  could  be  more  engaged  through 

Cabinet  Policy  Committees,  the  Agenda  and  Priorities 
Committee,  Treasury  Board  and  all-party  committees 
such  as  the  newly  formed  Policy  Field  Committees. 
The  media  were  interested  in  information  sessions 
when  PPRs  are  tabled. 

NGO  participants  demonstrated  a very  high  level  of 
understanding  of  public  performance  reporting 
concepts,  and  they  are  familiar  with  the  reporting 
practices  of  their  own  organizations.  This  suggests 
they  would  have  the  capacity  to  participate  effectively 
in  a stakeholder  engagement  process. 

Government  is  relying  more  than  ever  on  outside 
organizations  to  deliver  programs  and  services  that 
were  once  delivered  almost  exclusively  by 
government.  Many  participants  felt  that  PPRs  do  not 
adequately  reflect  this  shift;  i.e.,  such  organizations 
are  not  given  enough  opportunities  to  influence  the 
content  of  PPRs. 


( \ 

KEY  FINDINGS 

User  Engagement 

Participants  from  all  user  groups  said  the  quality  of  the  user 
engagement  process  directly  affects  the  credibility,  relevance, 
and  general  utility  of  PPRs.  They  valued  the  concept  of 
“mainstreaming"  public  performance  reporting,  i.e.,  the  idea  that 
a ministry  would  explicitly  review  and  discuss  goals, 
performance  measures,  targets,  and  reporting  mechanisms  with 
PPR  users  while  reviewing  existing  or  proposing  new  policies 
and  programs. 

Overview  of  CCAF  recommendations  related  to  each  of  the  key 
findings: 

♦ The  Government  of  Alberta  should  make  performance 
measures,  targets  and  reporting  mechanisms  an  integral 
and  explicit  part  of  consultations  with  stakeholders  on  policy 
development  and  program  delivery  issues,  in  order  to 
“mainstream”  public  performance  reporting.  This  could  be 
initiated  through  a pilot  project  with  a number  of  ministries. 

The  Ministry  of  Treasury  Board,  working  with  Executive 
Council,  could  advise  on  the  new  consultative  processes 
and  coordinate  the  sharing  of  best  practices,  thereby 
enhancing  integrated  ministry  planning  and  reporting 
capacity. 

♦ Ministries  should  seek  MLAs'  views  on  the  choice  of  goals, 
expected  outcomes,  strategies,  performance  measures, 
and  targets  when  officials  appear  before  legislative 
committees. 

♦ Ministries  should  hold  stakeholder  information  sessions  - 
either  in  person,  via  conference  call,  or  on-line  - when 
PPRs  are  tabled  in  order  to  discuss  the  results  of  the 
reports. 
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For  a complete  listing  of  the  recommendations , see  pages  7 to  25. 
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Project  Objectives 

Accountability  is  the  obligation  to  answer  for  how  one  discharges  one’s  responsibilities.  The 
manner,  style,  and  format  used  by  a government  to  communicate  and  account  to  its 
citizens  are  critical.  Reporting  must  be  targeted  toward  the  intended  audience  if  the 
information  is  to  be  understood  and  fully  used. 

As  part  of  its  commitment  to  be  accountable,  the  GoA  produces  annually  a variety  of  public 
performance  reports  (i.e. , business  plans,  budgets,  estimates  and  annual  reports).  For  the 
purpose  of  this  study,  public  performance  reports  (PPRs)  include  the  Government  of 
Alberta  business  plan,  ministry  business  plans,  the  Government  of  Alberta’s  annual  report 
Measuring  Up,  and  ministry  annual  reports.  The  financial  statements  in  these  documents 
were  excluded  from  review. 

Alberta  Premier  Ed  Stelmach  has  committed  to  making  the  Government  of  Alberta  more 
open  and  transparent.  Many  participants  in  these  consultations  welcomed  this  commitment 
and  believed  that  improving  PPRs  could  contribute  to  this  goal. 

The  objectives  of  this  project  are  to: 

♦ Further  develop  GoA  expertise  and  knowledge  in  assessing  the  needs  of  users  and  the 
uses  of  public  performance  information; 

♦ Strengthen  Alberta’s  leadership  role  in  government  accountability; 

♦ Improve  existing  plans  and  PPRs  or  develop  new  ones  that  better  meet  the  needs  of 
legislators,  the  media,  and  the  public;  and 

♦ Contribute  to  a larger  national  CCAF  study  on  PPRs. 

Key  Directions,  Findings  and  Recommendations 

The  key  directions  and  findings  that  emerged  during  the  consultations  have  been  organized 
into  four  themes: 

♦ Enhancing  the  credibility  of  PPRs; 

♦ Improving  the  relevance  of  the  goals,  strategies  and  performance  measures  found  in 
PPRs; 

♦ Increasing  the  clarity  and  accessibility  of  PPRs;  and 

♦ Engaging  stakeholders  in  Public  Performance  Reporting. 

These  directions  and  findings  are  presented  in  the  next  section. 
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WHAT  IS  CREDIBILITY?  - Credibility  is  a fundamental  quality  for 


PPRs.  For  the  Public  Sector  Accounting  Board  (PSAB),  the  more 
credible  a performance  report,  the  more  useful  it  is  to  its  users.1 
Credibility  is  an  attribute  that  users  assign  to  a report  based  on  their 
perceptions  and,  as  such,  can  be  subjective. 
The  Public  Performance  Reporting  Assessment  Guide2,  based  on 
PSAB  s Statement  of  Recommended  Practice  (SORP-2,  Public 
Performance  Reporting),  suggests  that  those  responsible  for  preparing 
and  reviewing  PPRs  evaluate  the  reports  features  to  identify  what 
information  is  included,  how  it  is  presented  and  interpreted,  and  what 
information  is  missing  that  could  influence  a users  perception  of  the 
report’s  credibility  and  its  usefulness.  Some  jurisdictions,  such  as  the 
Government  of  Canada,  have  issued  reporting  principles  based  on 
SORP-2  to  assist  report  producers  in  preparing  PPRs.3 


1 Slide  13,  Presentation  to  PPX  Forum,  entitled  The  Evolution  of  Public  Performance  Reporting,  http://www.ppx.ca/Symposium/ 
2007_symArchive/Presentations07/DayOne_PublicReportingE.pdf 

2 The  Guide  is  available  at:  http://www.psab-ccsp.ca/download.cfm?ci_id=37690&la_id=1&rejd=0 

3 Treasury  Board  Secretariat  issued  the  Government  of  Canada  Reporting  Principles  Crosswalk  to  the  CICA/PSAB/SORP-2: 
http://www.tbs-sct.gc.ca/rma/dpr3/06-07/principles-principes/princip_e.asp 
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Credibility 


Cause  and  Effect:  Measuring  Success  - Many  participants  observed  that  it  was  not 
always  clear  how  a policy  or  program  described  in  a public  performance  report  could 
achieve  its  stated  goal.  Generally,  participants  wanted  goals  and  performance  measures  to 
be  directly  relevant  to  the  quality  of  life  of  citizens. 

The  GoA  sometimes  measures  performance  based  on  whether  people  perceive  that  a 
result  has  been  achieved  (as  indicated  in  opinion  surveys).  The  use  of  these  “perception 
performance  measures”  (e.g.,  2006/07  Justice  Annual  Report,  Key  Measure  1)  produced 
mixed  reactions  across  participant  groups.  One  participant  asked:  “When  you  get  into 
performance  measures,  you  want  to  measure  the  end  results,  but  can  we  point  to 
something  the  government  has  achieved?  What  strategy  has  done  this?  The  government 
needs  to  be  accountable  for  what  they  did.”  A business  session  participant  asked:  “If 
government  is  measuring  their  departments’  success  by  high-level  results  like  provincial 
student  achievement  scores,  how  can  you  attribute  this  [a  higher  score]  to  a ministry  or  an 
employee’s  action?”  Only  the  media  wanted  more  perception  performance  measures,  such 
as  how  the  public  perceives  a program. 

Social/advocacy  NGOs  questioned  how  the  government  could  attribute  certain  results 
exclusively  to  the  government’s  action  and  not  to  a broader  mix  of  factors  in  which  the 
government  was  not  the  determining  force.  This  reaction  was  a result  of  the  goal-strategy- 
measure  structure  of  PPRs.  NGO’s  see  government  acting  more  and  more  as  a facilitator 
and  not  as  a direct  deliverer  of  services. 

One  business  participant  argued  that  it  was  disingenuous  to  claim  credit  for  factors  or 
results  beyond  the  government’s  control:  “Some  public  perception  questions  are  useful,  but 
not  things  like  safety  in  the  neighbourhood.  Can  the  provincial  government  address  this? 
What  about  municipal  impact  on  this?  Also,  this  doesn’t  reflect  the  differences  between 
cities  and  neighbourhoods.  The  provincial  government  should  report  on  things  it  can  do 
something  about,  such  as  recidivism  rates  and  crowding  in  jails,  etc.”  One  social/advocacy 
NGO  participant  said,  “Perceptions  can  drive  behaviours  and  people’s  sense  of  well  being. 
Actual  crime  figures  may  not  drive  behaviours.” 

In  contrast  to  participants’  perception  that  most  governments  are  too  willing  to  claim 
success  for  factors  beyond  their  control,  participants  across  user  groups  believe  that 
governments  are  unable  to  “admit  failure.”  This  creates  a perception  that  governments  are 
unable  or  unwilling  to  learn  from  their  mistakes.  One  social/advocacy  NGO  participant 
said,”  I don’t  expect  them  [governments]  to  succeed  at  all  times  [but]  how  do  you  find 
learnings  to  fix  things  if  there  is  no  identification  of  challenges?  Show  me  how  the 
government  can  reasonably  be  expected  to  impact  the  goal,  and  if  it  didn’t  work,  that’s  okay 
but  explain  why.” 

Admissions  in  PPRs  that  the  government  does  not  meet  certain  performance  measures  are 
seen  by  the  media  as  “false  humility.”  Social/advocacy  NGO  participants  noted  a “double 
standard”  - the  government  holds  not-for-profit  service  delivery  organizations  to  a higher 
standard  than  itself  when  reporting  on  results. 
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Need  For  Consistent  Performance  Measures  - If  longer-term  performance  measures 
are  to  be  used,  participants  wanted  the  performance  measures  data  (such  as  from  public 
opinion  surveys)  to  be  as  consistent  as  possible  from  year-to-year.  The  media  in  particular 
wanted  five  years  of  data  on  performance  measures  and  three  to  five  years  of  historical 
information  on  spending. 

Participants  also  asked  for  consistency  in  the  format  of  reports  across  ministries  and  from 
year  to  year.  They  asked  for  some  description  of  when,  why  and  how  performance 
measures  have  changed  from  year-to-year.  They  said  business  plans,  whose  format  is  set 
by  Treasury  Board,  are  more  usable  in  this  respect  than  annual  reports,  which  vary  more  in 
format  and  presentation.  A consistent  presentation  assists  the  media  to  find  and  report  the 
news  quickly. 

A Recommendation:  The  Government  of  Alberta  assume  a leadership  role  in  the 
development  of  performance  measures  for  horizontal  initiatives  (i.e.,  initiatives  in 
which  partners  from  two  or  more  organizations  agree  to  work  towards  the 
achievement  of  shared  outcomes). 


B Recommendation:  The  Ministry  of  Treasury  Board,  working  with  Executive 
Council,  assist  ministries  in  defining  and  articulating  the  ministry’s  strategic 
goals  and  performance  measures  so  that  users  of  public  performance  reports  can 
clearly  understand  how  a ministry’s  strategies,  programs  and  policies  contribute 
to  achieving  its  goals. 

C Recommendation:  The  Government  of  Alberta  develop,  update  regularly,  and 
make  publicly  available  guidance  to  ministries  on  the  principles  and  best  practices 
for  effective  public  performance  reporting  to  the  legislature.4 

D Recommendation:  The  Ministry  of  Treasury  Board  review  ministry  business 
plans  and  annual  reports  to  determine  their  level  of  compliance  with  Treasury 
Board  guidelines  regarding  the  principles  and  best  practices  of  effective  public 
performance  reporting.  In  doing  so,  the  Ministry  of  Treasury  Board  still  ensures 
that  ministries  maintain  ownership  of  their  PPRs. 


4 See  the  Appendix  for  a summary  of  the  Government  of  Canada  Treasury  Board  Secretariat’s  six  principles  for  effective  public 
performance  reporting. 
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Credibility  and  Impact  on  Perceptions  - Participants  from  all  the  consultation  groups 
believe  that  additional  information,  closer  alignment  between  goals,  strategies  and 
performance  measures,  as  well  as  changes  to  how  PPRs  are  prepared  could  improve  their 

perceptions  of  the  reports.  In  a representative  comment, 
one  participant  suggested  that  the  PPRs  “look  more  like  a 
political  statement  than  an  audit.” 

Several  participants  from  different  user  groups  singled  out 
the  bar  graph  on  page  44  of  the  GoA’s  Measuring  Up 
2006/07  Annual  Report  (see  insert  left).  They  questioned 
why  the  ministries  of  Sustainable  Resource  Development 
and  Tourism,  Parks,  Recreation  and  Culture  were  included; 
they  did  not  believe  these  ministries  had  any  involvement  in 
justice  issues.  The  reality  is  that  some  of  these  ministries’ 
programs  do  affect  the  results  of  that  goal.  (The  goal  states 
“Alberta  will  be  a fair  and  safe  place  to  work,  live  and  raise 
families.”)  The  ministries  were  listed  based  on  a national 
standard  under  which  government  expenses  are  classified 
by  a specific  function. 

Respondents  indicated  that  their  confidence  in  Alberta’s 
PPRs  would  increase  if  PPRs  had  independent  review  and 
input  (21.3%),  if  the  performance  measures/indicators  were 
seen  as  relevant  (19.1%),  and  if  there  was  consistency  in 
reporting  (14.9%). 

Longer  Time  Horizons  - Participants  agreed  that  in  a regular  three-year  government 
business  planning  cycle  or  the  regular  electoral  cycle,  it  is  hard  to  measure  longer-term 
outcomes.  All  consultation  participants  acknowledged  that  results  cannot  be  expected  from 
one  year  to  the  next  and  that  a concerted,  long-term  effort  is  in  fact  required.  The 
performance  measures  used  in  PPRs  should,  in  their  opinion,  reflect  this  reality.  One 
representative  from  the  Business  NGOs/professional  association  commented  “We  need  a 
whole  lot  more  information  on  the  experiences  or  results  of  government  spending,  which 
may  not  fit  in  the  longer-term  outcome  horizon.” 

E Recommendation:  The  performance  measures  used  in  PPRs  include  five  years  of 
historical  information  to  show  trends  and  ensure  consistency  for  year-to-year 
performance  measures. 

F Recommendation:  In  addition  to  the  high-level  longer-term  outcome  measures, 

PPRs  include  more  performance  measures  where  some  change  can  be  effected 
within  the  3-year  business  planning  cycle. 


“Goal  10:  Alberta  will  be  a fair  and  safe  place  to  work,  live  and 
raise  families.” 

Goal  Ten  Expense 


Percent  of  Total  Expense 

100 
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2005-06  Actual  2006-07  Budget  2006-07  Actual 

This  goal  includes  spending  of  the  following  ministries:  Immigration  and  Industry;  Finance; 
Infrastructure  and  Transportation;  Justice;  Municipal  Affairs  and  Housing;  Service  Alberta; 
Solicitor  General  and  Public  Security;  Sustainable  Resource  Development;  and  Tourism, 
Parks,  Recreation  and  Culture. 
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Use  of  Public  Opinion  Surveys  - In  addition  to  their  concerns  over  perception 
performance  measures,  participants  demonstrated  mixed  levels  of  trust  in  survey  numbers 
across  user  groups.  Generally,  trust  levels  increased  for  surveys  commissioned  by  a third 
party  instead  of  the  GoA.  Statistics  Canada,  for  example,  has  a high  level  of  credibility 
among  media.  Participants  valued  consistency  across  reporting  periods  for  survey  data, 
and  wanted  to  ensure  third  party  surveys  can  be  conducted  as  long  as  they  are  required. 

Clarity  - Participants  linked  the  writing  style  and  readability  of  PPRs  to  credibility.  Many 
suggested  that  PPR  authors  consider  using  a writing  style  aimed  at  the  general  public. 

The  survey  found  that  the  most  important  ranked  factor  that  would  increase  the  use  of 
Alberta’s  PPRs  was  if  “results  are  presented  in  a straightforward  and  simple  manner.”  This 
factor  was  also  ranked  highest  as  “Most  Important”  and 
“Important”  (see  Figure  below). 

Media  participants  suggested  that  a lack  of  clarity  in  the 
writing  creates  an  impression  that  government  is  “obscuring 
the  information.”  They  cited  page  44  of  Measuring  Up 
2006/07  (see  insert  right)  as  an  example  of  language  that 
would  leave  most  readers  confused.  They  suggested  that  “if 
you  make  it  easy  to  read  and  credible,  there  are  many 
others  who  would  use  it.”  Their  preference  is  for  a clear 
statement  of  the  existing  commitment,  followed  by  the 
result(s)  achieved  since  the  last  report,  and  the  target  to  be 
achieved  over  the  subsequent  reporting  period.  They 
wanted  such  a statement  to  be  complemented  by  a brief 
context,  rationale  and  analysis  section.  Given  the  complexity  of  PPRs,  the  media 
repeatedly  asked  that  government  experts  be  available  to  answer  technical  questions,  not 
necessarily  on  an  attribution  basis. 


/ N 

“To  address  the  government’s  top  priority  of 
Protecting  People’s  Private  Information,  the 
government  enhanced  its  principles,  policies, 
guidelines  and  practices  to  ensure  privacy  and 
security  of  the  personal  informationfor  which  it  is 
responsible.  A project  charter  and  tools  to  assist 
government  staff  in  the  management  and 
protection  of  personal  information  were 
developed 

- Measuring  Up  (page  44) 

\ ) 


What  would  increase  use  of  Alberts’s  PPRs?  If  results... 


Compare  Against  Other 
Results 

Include  Assmts.  of  Policy/ 
Program/Service  Areas 


Are  Current 

Are  Presented  in 
Straightforward/Simple  Manner 


■ Most  Important  Sllmportant  ■Somewhat  Important  H Least  Important 
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Some  participants  mentioned  the  GoA’s  Quarterly  Fiscal  Update  format,  where  planned  and 
actual  spending  is  presented  with  an  explanation  for  variances.  They  thought  this  format  of 
comparing  planned  versus  actual  results  could  be  applied  to  the  PPRs  where  reporting  on 
spending,  goals,  strategies  and  performance  increases. 

Inclusion  of  Financial  Data  - Some  participants  suggested  linking  business  plans  to 
annual  reports  and  budget  documents  so  that  program-level  spending  would  be  presented 
in  a more  transparent  manner.  They  also  believed  this  would  help  in  more  closely  aligning 
strategies,  goals  and  performance  measures.  “I  want  to  know  how  much  money  is  allocated 
to  these  areas,  and  if  spending  on  programs  is  going  up  or  down  over  time.  You’re  trying  to 
ascertain  this  beyond  the  language,  in  the  words,  using  key  information  to  exercise  some 
judgement  on  how  real  this  is,”  said  one  participant. 

Another  participant  in  that  group  wanted  “to  see  more  on  value  for  money  and  to  see  a 
correlation  between  the  Blue  Book5  and  Measuring  Up.  [...]  The  federal  department  lists 
grants  in  ministries  that  are  over  $25,000,  and  to  whom  it  [sic]  is  awarded.”  To  users, 
spending  levels  are  an  implicit  indicator  of  priority  that  help  users  determine  if  a program  or 
policy  is  adequately  funded. 

G Recommendation:  The  Government  of  Alberta  link  financial  information,  such 
as  budgets,  more  closely  to  non-financial  information,  such  as  strategies, 
performance  measures  and  targets. 

H Recommendation:  When  preparing  its  annual  report,  the  Government  of  Alberta 
compare  planned  versus  actual  spending,  list  what  goals,  strategies  and 
performance  measures  targets  were  met  or  not  met,  and  explain  why. 


Role  of  the  Office  of  the  Auditor  General  of  Alberta 
(OAG)  - A few  participants  expressed  concern  that  the 
recommendations  of  the  OAG  on  performance  measures 
are  not  addressed  by  government,  and  that  the  status  of 
government’s  implementation  of  the  recommendations  is 
unknown. 


s \ 

The  OAG  annually  completes  auditing  procedures 
on  the  performance  measures  and  results  included 
in  performance  reports.  An  audit  report  providing 
the  conclusions  and  any  exceptions  resulting  from 
this  work  is  included  in  Measuring  Up  and  each  of 
the  ministry  annual  reports.  Recommendations 
relating  to  performance  measures  are  made  to 
government  as  a result  of  this  work  and  separate 
systems  audits  performed  by  the  Office. 

V 


l Recommendation:  The  Government  of  Alberta  and  individual  ministries 
reference  relevant  Auditor  General  of  Alberta  recommendations  pertaining  to 
performance  measures  and  public  performance  reports  in  both  business  plans 
and  annual  reports,  and  indicate  the  status  of  their  implementation. 


5 The  Blue  Book  lists  payments  (e.g.  grants,  supplies  and  services)  made  to  individuals  and  organizations  by  the  Government 
of  Alberta. 
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WHAT  IS  RELEVANCE?  - CCAF  s nine  principles  of 


public  performance  reporting6  define  relevance  as 
“performance  information  [that]  relates  to  the  aspects  of 
performance  being  reported  on.  It  helps  users  to 
appreciate,  on  a timely  basis,  what  has  happened,  or  is 
likely  to  happen,  with  respect  to  those  aspects  of 
performance  that  are  seen  as  key  and  on  which  public 
reporting  is  focused.  It  explains  what  happened  and  why  it 
happened.  It  shows  whether  performance  is  improving  or 
declining  over  time  and  helps  predict  what  will  happen  in 
the  future.”7  Essentially,  relevance  is  the  degree  to  which  a 
performance  measure  is  pertinent,  inclusive,  timely,  and 

understandable. 

Relevance  is  a key  criterion  for  the  GoA  in  choosing 
performance  measures.  It  considers  a measure  to  have 
relevance  if  “the  measure  is  an  accurate  representation  of 
what  is  being  measured  and  the  information  presented  is 
timely  and  directly  related  to  the  subject  matter.”8 


6 The  2002  document  is  entitled  Reporting  Principles:  Taking  Public  Reporting  to  a New  Level.  An  executive  summary  is 
available  at  no  cost  at:  www.ccaf-fcvi.com/english/site_additions/archives2002/documents/executive_summary/ 
reporting_principles.pdf 

7 CCAF,  Reporting  Principles:  Taking  Public  Performance  Reporting  to  a New  Level,  pg.  45 

8 Measuring  Performance:  A Reference  Guide  is  available  at:  www.finance.alberta.ca/publications/measuring/measupgu/ 
pfmguide.pdf 
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Relevance 


Participants  found  the  structure  of  PPRs  - with  goals,  strategies,  and  performance 
measures  - useful.  That  being  said,  there  was  a wide  consensus  amongst  all  participants 
that  the  strategies  and  performance  measures  listed  in  the  reports  they  examined  could  be 
more  closely  aligned  to  goals.  They  questioned  the  causality  between  several  strategies 
and  goals.  Participants  from  the  media  and  NGO  groups  want  PPRs  to  place  greater 
emphasis  on  results  and  outcomes  instead  of  the  processes  used  to  achieve  the  results. 
They  also  agreed  that  performance  measures  should  reflect  the  influence  government 
actually  has  over  the  result  or  outcome. 


The  Consultation  Survey  found  that  the  Government  of  Alberta’s  Measuring  Up  Report  met 
respondents’  needs  the  least  (see  figure  below).  Measuring  Up  is  also  the  report  least  read 
by  respondents  at  71%  compared  to  the  most  read  Ministry  Business  Plans  (87.1%  of 
respondents). 


Percent  of  participants  reporting  PPRs  have  met  their  needs 


GOA  Strategic 
Business  Plan 


GOA  Measuring  Up 
Report 


Ministry  Business 
Plan(s) 


Ministry  Annual 
Report(s) 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


IS  Strongly  Agree  O Agree  □ Somewhat  Agree  ■ Somewhat  Disagree  IS  Disagree  □ Strongly  Disagree 


When  asked,  in  the  survey,  how  frequently  they  currently  use  PPRs  for  certain  purposes, 
65.5%  said  they  use  them  for  general  information,  49.1%  use  them  to  identify  priorities, 
46.3%  for  policy  development,  37.5%  to  assess  government’s  performance,  and  35.0% 
said  they  use  them  for  the  development  of  programs  and  services.  Only  21.9%  currently 
use  them  for  resource  allocation  decisions. 
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Providing  Context  - Many  participants  said  PPR  goals  and  performance  measures  often 
lack  information  to  assess  why  a target  was  set  at  the  level  given,  even  when  backed  by 
extensive  stakeholder  consultation.  This  was  the  case  when  one  group  of  participants 
questioned  why  one  measure  required  Alberta  employers  to  achieve  a workplace  lost-time 
claim  rate  of  2.0  which  was  selected  through  stakeholder  consultation.9  In  another  example, 
participants  did  not  know  why  one  goal  stated  that  Alberta  should  be  among  the  best  of  the 
four  western  provinces,  instead  of  among  other  peer  benchmarking  jurisdictions.10 

Media  participants  want  to  know:  “Why  was  the  target  set  where  it  was?  What  was  the 
methodology?  Why  was  the  target  not  met?  What  will  be  done  to  address  the  results?” 
When  reviewing  the  PPRs  provided  during  the  consultations,  participants  were  not  aware 
that  PPRs  included  the  methodology  of  public  opinion  surveys  or  other  contextual 
information. 

Enabling  users  to  seek  out  additional  context  and  background  is  much  easier  with  the 
online  version  of  PPRs,  where  hyperlinks  could  direct  readers  to  additional  information. 

J Recommendation:  The  Government  of  Alberta  review  the  technical  background 
information  currently  made  available  to  assist  users  in  better  understanding  the 
information  in  PPRs. 

K Recommendation:  All  PPRs  include  contact  information  for  senior  officials,  who 
can  provide  appropriate  background  technical  information. 


Linking  Annual  Reports  and  Business  Plans  - Many  participants  wanted  to  see  a 
closer  link  between  annual  reports  and  business  plans.  Media  participants  suggested  that 
PPRs  should  state:  a)  “here  is  what  we  said  we  were  going  to  do”  and  b)  “here  is  what  we 
did.”  One  participant,  wanting  PPRs  to  be  more  of  a “living  document,”  proposed  a “Fact- 
Assumption-Action”  framework.  In  this  proposal,  the  report  would  (i)  clearly  and  succinctly 
lay  out  what  the  existing  goal,  target  or  commitment  was  and  whether  it  had  been  achieved; 
(ii)  what  assumptions  were  embodied  by  the  goal;  and  (iii)  what  action  would  be  taken 
going  forward  to  achieve  the  new  target. 

L Recommendation:  The  Government  of  Alberta  improve  the  linkage  between 
business  plans  and  annual  reports  by  identifying  in  annual  reports  the 
assumptions  that  were  embodied  in  the  goals  and,  given  the  results,  what  actions 
would  be  taken  going  forward  to  achieve  the  goals. 


Role  of  the  OAG  — The  vast  majority  of  participants  across  user  groups  do  not  think  the 
OAG  should  be  responsible  for  a relevance  check  on  performance  measures.  Participants 
said  relevancy  strayed  into  policy  issues,  which  are  the  domain  of  the  government.  They 
also  think  that  the  OAG  does  not  have  the  skill  set  for  such  a role  as  the  OAG  had  not  been 
previously  called  upon  to  undertake  this  task. 


9 See  page  46  of  the  2006/07  Measuring  Up  Annual  Report. 

10  See  charts  on  page  44  of  the  2006  Measuring  Up  Annual  Report. 

* Providing  “Fairly  Presents”  Assurance  for  Performance  Reports:  A Methodology  Discussion  Paper  Focused  on  the  Relevance 
Assertion:  http://www.auditorroles.org/files/toolkit/role2/Tool2aBritishColumbia_RelevanceDiscussion.doc 
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WHAT  IS  CLARITY?  - Given  the  complexity  of  government 


programs  and  policies  and  their  relationship  to  goals  and  strategies,  it 
is  important  that  PPRs  be  as  clearly  written  as  possible.  The  broad 
range  of  readers  and  their  level  of  familiarity  with  public  performance 
reporting  makes  this  especially  difficult  to  achieve. 

In  its  guidelines  to  report  producers,  Treasury  Board  Secretariat  of 
Canada  advises  that  “for  performance  information  to  be  useful,  it 
must  be  capable  of  being  understood  by  users.  Explanatory  narratives 
should  be  precise  and  clearly  stated  in  plain,  non-technical  language 
that  focuses  on  critical  facts  and  matters  to  enable  users  to  obtain 
reasonable  insights  or  draw  reasonable  conclusions”11  This  is 
consistent  with  the  eighth  CCAF  principle,  that  public  performance 
reporting  “appropriately  embodies  the  characteristics  of  consistency, 
fairness,  relevance,  reliability  and,  most  especially, 

un  de  rstandability.” 1 2 

WHAT  IS  ACCESSIBILITY?  - For  the  purposes  of  this  report, 
accessibility  is  the  degree  to  which  the  underlying  data  contained  or 
referred  to  in  public  performance  reports  can  be  accessed,  parsed  and 
repurposed  by  users.  One  recent  example  that  embodies  this  concept 
of  accessibility  is  usaspending.gov,  a site  created  as  a result  of  the  US 
Federal  Funding  Accountability  and  Transparency  Act  of  2006. 13 


11  See:  http://www.tbs-sct.gc.ca/rma/dpr3/06-07/principles-principes/princip_e.asp 

12  Page  11:  http://www.ccaf-fcvi.com/english/site_additions/archives2002/documents/executive_summary/ 
reporting_principles.pdf 


13  The  Federal  Funding  Accountability  and  Transparency  Act  of  2006  (S.  2590)  requires  the  complete  disclosure  of  all 

organizations  receiving  federal  monies  on  a website  maintained  by  the  Office  of  Management  and  Budget.  The  site  makes  its 
data  available  in  an  Application  Programming  Interface  (API),  a source  code  interface  that  allows  for  a greater  degree  of 
automation  in  the  retrieval,  manipulation  and  repurposing  of  information.  The  web  site  is  at:  www.usaspending.gov/ 
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Clarity  and  Accessibility 


Including  Program-level  Statistics  and  Information  - Of  particular  interest  for 
participants  from  all  user  groups  was  the  inclusion  of  program-level  statistics  and 
information.  A few  social/advocacy  NGO  participants  believed  more  operational  details 
could  illustrate  results  and  provide  the  context  that  many  felt  was  missing.  As  one 
participant  put  it,  the  lack  of  operational-level  detail  in  PPRs  prevents  the  reports  from 
“tell[ing]  the  full  story”  and  reflecting  “on-the-ground  experiences.” 

M Recommendation:  The  Government  of  Alberta  include  more  program-level 
information  and  data  on  the  internet,  and  link  it  to  the  goals,  strategies  and 
performance  measures  found  in  the  business  plans  and  annual  reports. 

More  Data,  Less  Interpretation  - The  consultation  participants  said  PPRs  cannot  be  “all 
things  to  all  people.”  Relying  on  one  report  to  meet  the  needs  of  multiple  user  groups 
makes  the  reports  less  useful  to  all  and  places  an  unreasonable  demand  on  the  producers 
of  the  report. 


Instead  of  more  interpretation,  most  participants  want 
greater  access  to  government-held  data  (spending, 
demographics,  take  up/usage  rates,  etc.)  in  order  to 
enhance  their  planning  processes  and  program  delivery. 
They  believe  government  is  not  always  willing  or  able  to 
provide  greater  access. 

The  Consultation  Survey  found  that  participants  ranked 
“report  card/aggregated  indices  reports”  as  the  most 
preferred  content  for  Alberta’s  PPRs.  Participants  ranked 
both  “detailed/descriptive  reports”  and  “technical/data 
oriented  reports”  as  less  preferred.  However,  “technical/data 
oriented  reports”  were  ranked  as  second  “Most  Important”  to 
respondents  (see  figure  below).  The  survey  also  found  that  respondents  ranked  “graphs” 
as  the  presentation  format  they  would  most  prefer  to  see  in  Alberta’s  PPRs  and  “text”  as 
the  least  preferred  format. 


z'  \ 

Some  of  the  information  and  data  participants 
wanted  access  to,  either  through  PPRs  or  from 
other  sources,  fell  beyond  the  scope  of  this  report. 

For  example,  participants  requested  greater  access 
to  program-level  statistics,  which  falls  within  a 
broader  spectrum  of  uses.  As  any  decision  to  make 
such  information  available  would  have  an  impact 
on  how  users  perceive  PPRs,  the  report  includes  the 
feedback  of  participants  on  this  issue. 

\ / 
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Some  participants  want  government-wide  standards  for  data  to  ensure  that  apples-to- 
apples  comparisons  could  be  made  (e.g.,  how  an  “aging  population”  is  defined).  Other 
participants  noted  that  a more  automated,  technology-dependent  reporting  capacity  would 
require  enhanced  management  processes,  significant  resources,  and  additional  skills  within 
government.  A cost-benefit  analysis  could  help  determine  whether  and  how  to  move  to  a 
more  robust  online  reporting  capacity. 

Most  participants  wanted  greater  use  of  hyperlinks  in  online  versions  of  PPRs  so  users  can 
"drill  down”  for  more  detailed  or  contextual  information.  For  example,  a user  could  click  on  a 
measure  for  a description  of  why  it  is  relevant  to  the  goal,  how  it  was  agreed  upon  or  where 
to  find  further  details.  A graph  or  opinion  survey  could  also  be  hyperlinked  to  the 
methodology,  previous  data  series  or  changes  to  the  questions  asked.  For  print  versions, 
more  use  should  be  made  of  footnotes  or  endnotes  that  refer  readers  to  methodological 
information  for  performance  measures  and  public  opinion  surveys. 

N Recommendation:  Make  greater  use  of  hyperlinks  in  online  versions  of  PPRs  so 
users  can  “drill  down”  for  more  detailed  or  contextual  information. 

Layered  Approach  to  Reporting  - Participants  said  they  think  the  GoA  could  explore  a 
more  layered  approach  to  reporting.  This  was  echoed  in  the  Consultation  Survey.  To 
accomplish  this,  many  participants  proposed  better  harnessing  of  the  internet  and  currently 
available  technologies  when  considering  the  future  form  and  format  of  PPRs. 

All  user  groups  want  to  be  able  to  disaggregate  the  data  found  in  PPRs.  Some  participants 
want  data  broken  out  by  geography  (e.g.,  neighbourhood  or  constituency)  while  others 
think  more  demographic  and  socio  economic  data  would  be  valuable. 

There  was  a realization  that  some  data  would  lose  its  usefulness  when  disaggregated  as 
the  margin  of  error  would  increase  significantly.  Technology  could  assist  in  making 
disaggregated  data  available  online  so  that  readers  could  “draw  their  own  conclusions”  and 
easily  combine  public  performance  reporting  data  with  other  information  sources.  The  use 
of  hyperlinks  was  proposed  as  one  tool  to  enable  readers  to  drill  down  for  more  detailed 
information  in  the  online  version  of  PPRs. 

Other  participants  said  they  only  required  a high  level  of  information.  As  a useful  example 
of  a high-level,  marketing  oriented  document,  several  participants  cited  a recently  published 
brochure  prepared  by  the  Ministry  of  Employment,  Immigration  and  Industry  (Ell).14 

O Recommendation:  The  Ministry  of  Treasury  Board  review  how  other 

jurisdictions  are  using  new  technologies  to  enhance  their  PPRs  with  an  eye  to 
determining  which  might  be  suitable  in  Alberta,  given  resource  issues. 

Frequency  of  Reporting  - Reactions  were  mixed  across  all  user  groups  as  to  whether  all 
PPRs  should  be  released  on  the  same  day  or  if  a more  staggered  release  should  be 
considered  (i.e.,  annual  release  but  with  half  of  the  reporting  organizations  disclosing  on 


14  For  Ell  publications,  visit  the  ministry’s  web  site  at:  http://employment.  alberta.ca 


IMPROVING  PUBLIC  PERFORMANCE  REPORTS 


19 


one  day  and  the  other  half  on  another  day).  The  media,  in  particular,  found  it  difficult  to 
review  PPRs  when  they  were  all  released  on  the  same  day. 

Some  participants  noted  that  the  OAG  will  be  moving  to  a semi-annual  reporting  schedule. 
They  suggested  that  the  Ministry  of  Treasury  Board  should  monitor  this  change  to  help  it 
assess  whether  a more  staggered  release  should  be  considered  for  PPRs. 

As  for  whether  each  reporting  organization  (e.g.,  ministry)  should  report  more  than  once 
each  fiscal  year,  participants  did  not  reach  a consensus.  One  participant  suggested  that 
more  frequent  reporting  would  make  a PPR  more  of  a “living  document”  and  more  useful  to 
managers  and  decision-makers  within  government.  Other  participants  questioned  whether 
legislated  requirements  prevent  a staggered  release. 

PPR  Portal  - Participants  proposed  creating  a central  website  or  portal  for  PPRs  as  is 
done  with  business  plans.15  Media  participants  wanted  both  printed  copies  and  an  ability  to 
easily  access  documents  online. 

P Recommendation:  The  Government  of  Alberta  create  a single  online  web  portal 
where  users  could  access  all  PPRs  (including  those  from  previous  years)through 
improved  search  functions.  The  URL  for  this  portal  should  be  included  in  all 
communications  materials  dealing  with  PPRs. 


15  See:  http://www.finance.alberta.ca/publications/bus_plans/index_all.html 


20 


IMPROVING  PUBLIC  PERFORMANCE  REPORTS 


Engaging  Users 


WHAT  IS  ENGAGEMENT?  - The  consultations  suggested  that 
users’  perceptions  of  PPRs  are  directly  influenced  by  how  much  and 
how  well  they  are  able  to  engage  in  the  formulation  of  the  PPRs  and  in 
any  dialogue  that  may  occur  after  the  publication  of  PPRs.  This  link 
was  confirmed  by  participants’  responses  to  the  Consultation  Survey, 
in  which  they  list  “independent  review/input”  as  the  choice  that 
would  most  increase  their  confidence  in  PPRs.  Moreover,  if  users  were 
not  engaged  in  the  public  performance  reporting  aspects  of  a policy  or 
program,  they  tended  to  have  a low  level  of  familiarity  with  many  of 
the  PPR  performance  measures  and  their  relationship  to  policy  or 
program  objectives.  The  context  and  rationales  for  the  performance 
measures  were  repeatedly  questioned.  This  situation  served  neither 
the  producers,  who  want  these  reports  to  be  meaningful  and  useful, 
nor  the  users,  who  would  make  greater  use  of  PPRs  if  the  reports 

better  met  their  needs. 

WHAT  IS  MAINSTREAMING?  -"Mainstreaming"  public 
performance  reporting  occurs  when  a ministry  explicitly  reviews,  and 
discusses  with  stakeholders,  the  policy  or  program  objectives, 
performance  measures,  targets,  and  reporting  mechanisms  in 
conjunction  with  its  review  of  existing  policies  or  prgrams 


or  when  proposing  new  policies  and  programs. 


Engaging  users 


Mainstreaming  Performance  Measurement  - Participants,  particularly  from  business 
NGOs/professional  associations  and  social/advocacy  NGOs,  noted  how  the  public  policy 
development  process  and  program  delivery  arrangements  have  dramatically  changed  over 
the  last  decade.  Stakeholders  are  being  called  upon  to  provide  input  on  a wide  range  of 
policy  and  program  areas.  More  than  ever,  government  is  relying  on  outside  organizations 
to  deliver  programs  and  services  once  delivered  almost  exclusively  by  government.  To 
ensure  accountability,  reporting  and  performance  arrangements  have  evolved.  These 
arrangements  are  usually  spelled  out  in  service  contracts  between  the  government  and  the 
service  provider. 

Participants  believe  they  could  provide  valuable  input  on  performance  measures,  targets, 
and  reporting  mechanisms  when  programs  are  being  developed  instead  of  after  the  fact. 
They  believed  their  involvement  would  enable  a closer  alignment  of  the  goals,  strategies 
and  performance  measures  found  in  PPRs. 

Although  many  ministries  do  consult  stakeholders  about  performance  measures  when 
discussing  policies  or  programs,  these  discussions  tend  to  be  implicit  rather  than  explicit. 
Also,  when  they  do  occur,  they  usually  happen  at  the  operational  level  and  the  results  are 
not,  for  the  most  part,  fed  into  the  public  performance  reporting  process.  This  created  the 
impression  among  marry  participants  that  public  performance  reporting  is  a stand  alone 
process,  separate  from  and  not  involving  these  outside  user  groups. 

Participants  felt  that  an  enhanced  and  more  explicit  dialogue  between  users  and  producers 
of  public  performance  reporting  could  be  the  basis  for  a new  longer-term  relationship.  As 
one  NGO  participant  put  it,  “Some  industries  are  specific  or  can  count  on  more  black  and 
white  measures,  like  births,  graduation  numbers,  etc.  But  when  you  are  measuring  complex 
social  change,  we  need  to  work  together  on  this.  None  of  us  has  the  answer,  but  [working 
together]  in  the  right  environment  could  be  very  productive.” 

CCAF  suggests  that  three  or  four  ministries  that  already  have  a strong  culture  of 
collaboration  with  outside  stakeholders  could  participate  in  such  a dialogue  (an  alternative 
would  be  a single  multi-ministry  business  area).  It  would  be  prudent  to  ensure  that  the 
selected  policy  area  has  broad  support  across  all  political  parties.  This  will  help 
“mainstream”  public  performance  reporting  on  a go-forward  basis.  When  ministries  review 
existing  policies  and  programs,  they  should  also  review  the  goals,  performance  measures, 
targets,  and  reporting  mechanisms  in  order  to  “mainstream”  public  performance  reporting 
for  existing  programs. 

The  exact  modalities  of  a public  performance  reporting  consultative  process  are  likely 
different  from  ministry  to  ministry  given  differing  organizational  cultures,  stakeholders,  and 
mandates.  The  Ministry  of  Treasury  Board,  working  with  Executive  Council,  could  advise  on 
consultative  processes,  coordinate  the  sharing  of  best  practices  thereby  enhancing  ministry 
capacity.  This  would  drive  improvements  across  the  GoA,  as  it  pertains  to  integrated 
planning  and  reporting  processes. 

“Champions”  within  government  would  be  needed  for  successfully  piloting  a new  approach 
to  PPRs.  High-level  ministry  staff  would  need  to  support  this  initiative  and  public  servants 
would  also  require  political  support  from  the  Premier  and  Cabinet  Ministers. 
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In  addition,  many  participants  requested  information  sessions  with  ministry  officials  upon 
the  tabling  of  PPRs  to  discuss  the  reports.  These  sessions  could  take  the  form  of  in-person 
technical  briefings,  such  as  those  provided  at  the  budget  lockup,  or  could  be  via  conference 
call  or  online,  depending  on  the  level  of  stakeholder  interest. 

Determining  Relevance  Through  User  Engagement  - Most  participants  agreed  that 
there  is  a role  for  an  “external  party"  to  comment  on  the  relevance  of  performance 
measures.  There  was  little  consensus,  either  within  or  across  user  groups,  as  to  how  that 
would  be  accomplished. 

A minority  of  the  participants  believe  that  an  external  check  might  be  more  valuable  as  a 
management  tool  as  a way  to  engage  users.  Social/advocacy  NGO  participants  want 
organizations  affected  by  the  performance  measures  in  PPRs  to  be  included  in  determining 
relevance. 

This  was  echoed  in  the  Consultation  Surveys  where  one  respondent  wrote,  “The  criteria 
and  performance  measures  need  to  be  set  in  consultation  with  the  public  and  key 
stakeholders  who  are  affected  by  the  policies.  Once  these  performance  measures  are  set 
they  need  to  be  produced  by  the  departments  in  ways  that  show  how  we  are  doing  in  each 
area.”  Their  reasoning  is  based  on  their  experience  that  existing  performance  measures 
“are  all  quantitative  measures,  but  there  is  no  way  to  capture  qualitative  information.  These 
numbers  may  not  reflect  the  ‘on  the  ground’  experience.”  Many  said  that  determining 
relevance  was  more  of  a collaborative,  government-led  process  than  one  led  from 
elsewhere. 

The  users  who  were  consulted  are  very  interested  in  contributing  to  the  process  of  setting 
performance  measures  and  targets,  and  in  providing  input  on  reporting  mechanisms.  Many 
also  expressed  an  interest  in  receiving  an  information  briefing  when  the  PPRs  are  tabled  in 
the  legislature.  An  enhanced  dialogue  between  users  and  producers  could  be  the  basis  for 
a new  longer-term  relationship. 

Q Recommendation:  Future  stakeholder  consultations  related  to  new  policies  and 
programs  include  an  explicit  discussion  of  performance  measures,  targets  and 
reporting  mechanisms  and  how  frequently  performance  measures  and  targets 
should  be  reviewed. 

R Recommendation:  Undertake  pilot  project  approach  to  public  performance 
reporting  consultations. 

S Recommendation:  Common  principles  - such  as  a requirement  to  consult  a 
representative  group  of  stakeholders  - applied  across  ministries. 

T Recommendation:  The  Government  of  Alberta  identify  “champions”  within 
government  to  successfully  implement  new  approaches  to  PPRs. 

U Recommendation:  Ministries  hold  information  sessions  for  stakeholder  user 
groups  when  PPRs  are  tabled  in  order  to  discuss  the  results  of  the  reports. 
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Role  of  the  Legislature -According  to  governance  expert  Robert  Marleau,  the  former 
Clerk  of  the  House  of  Commons,  a legislature  “has  two  basic  roles,  [. . .]  that  is  to  pass 
legislation  and  supply.  It  has,  in  my  view,  over-focussed  on  legislation  in  the  last  25  years 
and  almost  abandoned  its  constitutional  responsibility  on  supply.”16 

PPRs  are  a key  tool  for  a legislature  to  conduct  its  supply  role  and  to  hold  the  government 
of  the  day  to  account  for  the  effective  spending  of  public  monies,  including  the 
government’s  performance  in  achieving  the  objectives  it  sets  for  itself.  CCAF  research 
suggests  that  much  of  the  initial  reforms  to  public  performance  reporting  focused  on  the 
quality  of  the  financial  information  available  to  the  legislative  branch.  This  consultation 
project  reinforced  the  conclusion  that  the  current  focus  of  reforms  is  moving  from 
governmental  outputs  (e.g.,  clients  served,  birth  certificates  issued)  to  higher-level 
outcomes  that  demonstrate  how  government’s  actions  impact  citizens. 

There  are  many  opportunities  for  engaging  of  MLA’s  in  dialogue  on  goals,  expected 
outcomes,  strategies,  performance  measures  and  targets.  For  example,  when  officials 
appear  before  committees,  such  as  Cabinet  Policy  Committees,  the  Agenda  and  Priorities 
Committee,  Treasury  Board  and  all-party  committees  like  the  newly  formed  Policy  Field 
Committees,  or  on  other  occasions. 


V Recommendation:  Ministries  make  a concerted  effort  to  engage  MLA’s  in 
dialogue  on  goals,  expected  outcomes,  strategies,  performance  measures  and 
targets. 

Role  of  All-party  Committees  - Many  participants  believe  that  the  Legislative  Assembly 
of  Alberta  and  its  elected  members  have  an  important  role  to  play  in  public  performance 
reporting  and  in  holding  the  government  to  account  for  the  effective  spending  of  public 
monies.  These  participants  also  believe  that  all-party  committees  should  play  a role  in 
reviewing  and  commenting  on  PPRs.  Most  participants,  except  the  legislators,  were 
unaware  that  the  Policy  Field  Committees  (PFCs)  are  already  mandated  to  review  ministry 
annual  reports. 

MLAs  reflected  on  their  experience  with  the  recently  constituted  PFCs,  seeing  them  as  a 
“work  in  progress.”  Based  on  this  experience,  they  indicated  that  the  degree  to  which  PFCs 
could  serve  as  effective  review  committees  would  be  dependent  on  the  degree  to  which 
their  work  would  not  be  duplicated  by  the  legislature  sitting  as  a Committee  of  the  Whole. 
They  also  agreed  that  PFCs  will  continue  to  evolve  as  will  the  culture  of  the  committees. 
Most  participants,  including  MLAs,  believe  that  since  the  PFCs  were  newly  formed  and 
have  been  focused  primarily  on  discussing  new  legislation,  it  would  take  time  before  they 
would  review  performance  reports.  Some  participants  anticipated  that  PFCs  would  have  a 
very  heavy  workload  reviewing  bills  and  questioned  whether  reviewing  PPRs  would  be  the 
most  effective  use  of  the  committees’  time. 


16  The  Hill  Times.  No.  625,  Legislative  Process,  February  18,  2002.  Robert  Marleau  was  the  Clerk  of  the  House  of  Commons 
from  1987  to  January  2001. 
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A few  participants  observed  that  the  PFCs  might  be  able  to  serve  in  an  oversight/review 
role  once  a policy  or  program  is  implemented  and  the  performance  measures,  targets,  and 
reporting  mechanism  are  already  in  place.  Stakeholders  could  provide  independent 
feedback  via  ministries,  before  Policy  Field  Committee  review. 

Participants  expressed  a desire  for  an  external  forum  to  review  business  plan  goals,  the 
strategies  adopted  by  the  government  to  achieve  them  and  the  performance  measures 
chosen  to  determine  effectiveness. 

Two  existing  practices  from  different  Canadian  jurisdictions  featuring  legislative  involvement 
in  public  performance  reporting  are  described  in  Appendix  IV  of  this  report. 

W Recommendation:  Policy  Field  Committees  be  the  primary  forum  to  review 
stakeholder  opinions  about  performance  measures,  targets,  and  reporting 
mechanisms  as  part  of  their  mandate  to  review  annual  reports. 
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Participant  Groups 

The  next  section  of  this  report  provides  additional  detail  on  the  comments  made  during  the 
consultation  by  the  individual  user  groups. 

MEDIA 

Although  not  all  the  media  participants  use  performance  reports,  those  who  do  use  them  for 
similar  purposes.  In  the  Consultation  Survey,  a higher  percent  of  media  participants 
reported  they  have  not  read  Ministry  Business  Plans  or  Government  of  Alberta  (GoA) 
Strategic  Business  Plan,  when  compared  to  the  other  user  groups.  Most  stated  that  they 
used  PPRs  to  “compare  political  rhetoric  to  what  actually  happened,”  often  referring  to  them 
well  after  they  had  been  issued.  The  participants  estimated  that  they  refer  to  GoA  business 
plans  or  annual  reports,  on  average  two  to  three  times  a year.  Overall,  they  view  PPRs  as  a 
“valuable  piece  of  reference,  even  if  rarely  used.”  They  find  PPRs  often  contain  interesting 
information  on  the  state  of  Alberta’s  economy  and  society.  That  being  said,  they  also  want 
operational  details  included  in  the  reports  to  make  them  more  relevant.  They  also  felt  there 
was  a need  for  more  plain  language  in  the  way  the  reports  are  written. 

Most  find  it  difficult  to  digest  “2000-plus  pages  worth  of  information,”  and  they  said  that 
PPRs  make  for  "terrible”  TV  coverage.  They  suggest  the  government  hold  a technical 
briefing  when  ministry  annual  reports  are  released  and  include  the  name  and  contact 
information  for  senior  officials  that  could  provide  technical  background.  Participants  thought 
it  would  be  worthwhile  to  have  a portal  where  all  reports  could  be  accessed. 

Most  media  participants  indicated  they  are  looking  for  trends,  usually  on  a five-year 
timeline.  They  said  it  is  important  for  data  sets  in  PPRs  to  reflect  this.  They  often  use  this 
information  in  graphical  illustrations.  Most  indicated  they  want  more  consistency  for  year-to- 
year  performance  measures.  If  this  was  not  possible,  or  if  there  were  changes  to 
performance  measures,  then  they  would  like  changes  to  be  noted  and  an  explanation 
provided.  Otherwise,  it  could  appear  that  officials  changed  performance  measures  so  that 
lacklustre  results  could  not  be  tracked. 

Despite  their  perception  that  PPRs  were  of  an  overly  “political  nature,”  media  participants 
took  this  consultation  seriously  and  hoped  for  incremental  improvement.  They 
acknowledged  the  difficulty  in  producing  PPRs  that  completely  met  their  needs. 

BUSINESS  NGOs  AND  PROFESSIONAL  ASSOCIATIONS 

Generally,  these  participants  rely  on  ministry  business  plans  more  than  annual  reports. 
Business  plans  are  seen  to  be  useful  to  track  government  planning  and  spending.  The 
participants  also  find  that  budget  documents  are  useful. 

Business  participants  used  the  reports  for  a broader  range  of  uses  than  media  participants. 
They  tend  to  focus  on  the  ministry  reports  that  directly  affect  their  organizational  interests. 
“We  use  reports  from  a ministry  depending  on  whether  it  affects  our  members,”  said  one 
participant. 

Another  participant  described  how  he  uses  the  reports  to  determine  points  of  commonality 
and  to  align  his  organization’s  plans  with  the  government’s  plans.  Another  uses  them  to 
track  spending  levels  in  order  to  determine  if  a priority  is  adequately  funded. 
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Other  uses  of  PPRs  by  this  group  included: 

♦ To  determine  policy  trends  in  the  government; 

♦ To  promote  the  interests  of  the  organization’s  members; 

♦ For  government  relations  activities; 

♦ To  anticipate  legislative  change; 

♦ To  research  an  organization’s  field  of  activity; 

♦ For  teaching/educational  purposes;  and 

♦ As  background  information  for  collective  bargaining. 

Most  participants  see  PPRs  as  public  relations  documents  for  the  government.  Some 
participants  believe  the  documents  are  not  as  detailed  as  they  should  be  and  that  they 
require  readers  to  consult  other  sources  of  information. 

This  group  also  said  that  in  many  cases  they  could  not  attribute  the  results  to  government 
action,  which  furthered  the  appearance  that  PPRs  are  public  relations  documents.  That  is, 
they  felt  PPRs  dealt  with  goals  or  performance  measures  that  government  had  little  control 
or  influence  over.  They  also  wanted  reports  placed  on  a web  site  with  more  explanatory 
information  available  regarding  why  a performance  measure  and  a target  were  chosen,  as 
well  as  the  ability  to  drill  down  to  more  detailed  data.  A few  participants  said  a mechanism 
for  an  all-party  review  of  PPRs  would  aid  in  improving  the  credibility  of  the  documents. 

A number  of  participants  referred  to  Employment,  Immigration  and  Industry’s  publications, 
specifically  this  ministry’s  brochure  listing  its  priorities  and  lead  officials.  They  wanted  a 
contact  person  to  be  able  to  call  in  the  ministries  for  more  information. 

Some  participants  mentioned  the  GoA’s  Quarterly  Fiscal  Update  format,  where  planned  and 
actual  spending  is  presented  with  an  explanation  for  variances.  They  thought  this  format  of 
comparing  planned  versus  actual  results  could  be  applied  to  the  PPRs  when  reporting  on 
spending,  goals,  strategies  and  performance  measures.  They  also  wanted  more  integration 
of  financial  and  non-financial  information,  where  value-for-money  information  would  be 
presented. 

SOCIAL/ ADVOCACY  NGOs 

These  participants  rely  on  PPRs  for  a broad  spectrum  of  uses.  As  with  many  participants 
across  different  user  groups,  these  participants  focus  more  on  business  plans  and  how 
funds  are  allocated  than  on  annual  reports.  “Business  plans  are  informative  and  are 
communicated  well,”  said  one  participant.  This  was  in  line  with  the  opinions  of  other  user 
groups.  The  government’s  business  plans  are  more  useful  for  their  planning  purposes. 
Again,  participants  believe  that  the  audience  for  PPRs,  particularly  annual  reports,  is  not 
clearly  defined.  “In  any  document  like  this,  you  can’t  shoot  for  the  middle  - then  it  doesn’t 
work  for  anyone.  For  example,  if  this  is  for  the  NGO  sector,  we  understand  this  stuff  and 
want  more  rigorous  information,  yet  for  the  public,  this  information  and  these  performance 
measures  may  be  enough,”  said  one  participant. 

Some  use  them  (together  with  information  exchanges  with  ministry  contacts)  to  assess 
government  direction.  Others  use  them  to  refine  the  design  of  their  organization’s  service 
plan,  identify  service  gaps  that  may  exist,  to  determine  if  their  organization  could  and 
should  address  those  service  gaps,  and  to  better  align  their  existing  services  with  others  in 
their  “space.”  Several  rely  on  PPRs  for  internal  strategic  planning  processes  and  contextual 
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information.  Many  use  them  to  assess  government  priorities  and  to  match  them  against 
their  own  organizational  priorities.  PPRs  are  useful  in  determining  the  GoA’s  level  of 
understanding  of  “what  goes  on”  in  the  non-profit  sector  and  in  educating  the  government 
about  the  sector,  including  underlying  systemic  or  societal  issues.  Umbrella  organizations 
placed  the  reports  in  their  library  for  their  members  to  use.  Several  of  these  uses  were  also 
captured  in  the  Consultation  Survey. 

Participants  from  this  group  are  very  aware  of  how  the  policy  process  had  become  much 
more  diffuse  and  how  government  and  NGOs  now  work  much  more  in  collaboration  in 
order  to  achieve  results.  They  believe  that  the  public  performance  reporting  process  has 
not  changed  in  line  with  these  other  changes,  and  that  this  disconnect  negatively  influences 
their  perceptions  of  and  attitudes  towards  PPRs. 

One  respondent  noted  that  the  level  of  detail  and  specificity  in  the  government  reports 
would  be  deemed  inadequate  for  a funded  grant  organization.  “Contracted  agencies  are 
told  that  they  have  to  prove  results,  but  government  doesn’t  have  to  do  so,”  said  one 
participant.  Another  participant  stated,  “We  look  at  them  for  what  the  reports  are  not  saying. 
If  you  read  the  other  ministry  reports,  there  are  so  many  things  government  says  they  want 
to  do,  but  then  the  reports  don’t  measure  these  things.” 

Another  participant  stated  that  although  PPRs  are  tied  to  the  budget,  there  should  be  an 
identifiable  link  between  priorities,  programs,  policies  and  their  budgets  and  that  financial 
data  should  be  relative  to  population  growth  (e.g.,  per  capita)  or  as  a percentage  of  total 
GDP,  as  these  benchmarks  would  allow  users  to  compare  Alberta  with  other  provinces. 

Legislators 

MLAs  generally  made  limited  use  of  PPRs,  unless  they  received  questions  from 
constituents  or  serve  on  a Cabinet  Policy  Committee.  Government  MLAs  identified  a 
number  of  places  where  they  have  an  opportunity  to  be  involved  in  reviewing  and  using 
PPRs.  They  listed  the  Cabinet  Policy  Committees,  the  Policy  Field  Committees  (PFCs),  the 
Agenda  and  Priorities  committee  and  Treasury  Board  committee,  where  PPRs  could  be 
reviewed  before  they  are  published.  They  noted  that  the  Public  Accounts  Committee  could 
review  PPRs  after  they  are  published.  Some  legislative  committees,  such  as  the  Public 
Accounts  Committee,  do  make  use  of  PPRs,  particularly  annual  reports. 

MLAs  are  open  to  playing  a greater  role  in  public  performance  reporting  through  PFCs  but 
they  said  this  would  need  to  be  balanced  against  other  priorities.  MLAs  reflected  on  their 
experience  with  the  recently  constituted  PFCs,  seeing  them  as  a “work  in  progress.”  They 
expect  the  committees  will  have  a growing  workload,  centred  on  reviewing  legislation.  They 
agreed  that  PFCs  will  continue  to  evolve  as  will  the  culture  of  the  committees.  Opposition 
MLAs  found  the  work  to  date  of  the  PFCs  to  be  more  worthwhile  than  government  MLAs. 

MLAs  want  to  see  more  outcome-based  performance  measures  which  measure  a wider 
range  of  subject  areas.  They  believed  more  disaggregated  data,  particularly  that  broken 
down  by  riding,  would  be  of  use  to  them  and  their  electorate  who  focus  on  local  issues.  For 
those  who  do  legislative  committee  work,  they  required  information  presented  by  specific 
issue,  and  concise  and  tailored  for  short  meetings.  They  also  want  more  timely  information 
and  suggested  quarterly  updates  (but  not  more  reports)  might  be  more  useful  for  some  of 
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their  purposes  than  an  annual  cycle.  Legislators  spoke  about  the  need  for  layered  reports 
to  meet  specific  purposes. 

MLAs  do  not  think  it  should  be  the  role  of  the  OAG  to  provide  a relevance  check  for 
performance  measures.  Some  believe  greater  stakeholder  input  could  address  many  of  the 
issues  related  to  relevance  raised  in  this  consultation  process,  but  the  costs  of  collecting 
the  data  for  recommended  performance  measures  would  have  to  be  considered. 

INDIVIDUAL  INTERVIEWS 

One  interviewee  spoke  of  his  need  for  comprehensive  and  timely  information  so  that  he 
and  his  executive  teams  can  make  the  most  informed  decisions  possible.  Generally,  he  and 
other  interviewees  do  not  use  PPRs  on  a day-to-day  basis.  PPRs  are  the  foundation  of  the 
annual  policy  planning  cycle  that  starts  with  an  annual  policy  retreat.  The  interviewees 
found  that  strong  management  practices  need  to  underpin  reporting  practices  so  that  the 
right  information  can  be  captured  and  brought  to  bear  on  the  issues  of  the  day. 

The  interviewees  consider  the  PFCs  a “work  in  progress”  and  expect  that  PFCs  will  have  a 
growing  workload  focused  on  reviewing  legislation.  Some  interviewees  do  not  believe  that 
reviewing  PPRs  would  be  the  most  effective  use  of  the  PFCs’  time  at  the  outset.  The  PFCs 
would  need  to  create  a working  culture  where  the  public  interest  could  be  placed  ahead  of 
political  ones  and  where  a spirit  of  constructive  partisanship  could  emerge.  These 
interviewees  anticipate  that  if  performance  measures  and  reporting  are  addressed  during 
the  policy  development  stage,  there  would  be  less  need  for  the  PFCs  to  tackle  that  work. 
Several  interviewees  say  stakeholders’  expectations  would  need  to  be  managed  if  the 
process  were  opened  up,  as  the  GoA  would  ultimately  need  to  decide  which  targets  to  use 
in  managing  programs. 

One  interviewee  noted  the  importance  of  government-wide  data  standards  in  public 
performance  reporting,  such  as  references  to  specific  demographics  (i.e.,  aging  population) 
or  how  a calculation  is  completed  (high-school  completion  rates).  Such  standards  would 
assist  in  tracking  and  reporting  across  ministries.  “One  thing  government  can  do  is  ensure 
government  is  reporting  the  same  facts.  I’m  not  sure  that  we’re  there  today,”  said  the 
interviewee. 

Most  interviewees  want  to  see  a greater  link  between  business  plans  and  annual  reports. 
One  interviewee  with  knowledge  of  Australia’s  PPR  practices  noted,  “Australia  did  an 
overall,  more  strategic  document,  and  there  were  more  detailed,  operational  documents. 
These  were  linked.”  Most  interviewees  say  this  would  enhance  management  practices  and 
increase  transparency.  One  interviewee  proposed  a Fact-Assumption-Action  framework 
and  suggested  that  PPRs  become  more  of  a “living  document”  where  the  report  would 
(i)  clearly  and  succinctly  lay  out  what  the  existing  goal,  target  or  commitment  is  and 
whether  it  has  been  achieved;  (ii)  what  assumptions  are  embodied  in  the  goal;  and  (iii)  what 
action  will  be  taken  going  forward  to  achieve  the  new  target.  The  clarity  of  the  language 
used  in  the  reports  was  also  noted.  “Ministry  reporting  needs  clearer  writing  - the  reports 
are  difficult  to  read.  The  Australian  report  was  easier  to  read  because  of  the  language  they 
used,”  noted  the  interviewee  with  knowledge  of  Australian  PPR  practices. 
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Recommendations  and  Proposed  Next  Steps 


The  following  criteria  were  used  when  considering  whether  a recommendation  could  be 
implemented  in  the  short-term  (six  months  to  one  year),  medium-term  (one  to  two  years),  or 
longer-term  (two  or  more  years): 


♦ The  degree  to  which  the  recommendation  is  currently  already  addressed  by  the 
Government  of  Alberta. 

♦ Whether  implementation  of  the  recommendation  requires  changes  to  legislation  or 
regulations,  thereby  extending  the  time  to  implementation. 

♦ The  cost  of  implementing  the  recommendation.  The  high  cost  of  implementing  a 
recommendation  would  result  in  a more  complex  and  longer  review  and  approval 
process. 

♦ The  cost  of  implementing  the  recommendation  relative  to  the  benefit. 

Recommendations  with  a higher  benefit  relative  to  the  cost  could  be  justified  for  an  early 
implementation. 

♦ The  organizational  complexity  of  implementing  the  recommendation. 

Recommendations  that  require  the  cooperation  of  many  ministries,  and  or  external 
stakeholders  would  require  more  time  to  organize  and  implement. 

♦ The  technical  complexity  of  supporting  systems  required  to  fully  implement  a 
recommendation.  More  complex  supporting  technical  systems  require  time  to  obtain  the 
technical  resources  and  to  develop  and  implement  the  required  systems. 

Short-term 


P Recommendation:  The  Government  of  Alberta  create  a single  online  web  portal 
where  users  could  access  all  PPRs  (including  those  from  previous  years)  through 
improved  search  functions.  The  URL  for  this  portal  should  be  included  in  all 
communications  materials  dealing  with  PPRs.  (Clarity  and  Accessibility) 

K Recommendation:  All  PPRs  include  contact  information  for  senior  officials,  who 
can  provide  appropriate  background  technical  information.  (Relevance) 

0 Recommendation:  The  Ministry  of  Treasury  Board  review  how  other 
jurisdictions  are  using  new  technologies  to  enhance  their  PPRs  with  an  eye  to 
determining  which  might  be  suitable  in  Alberta,  given  resource  issues.  (Clarity 
and  Accessibility) 

1 Recommendation:  The  Government  of  Alberta  and  individual  ministries 
reference  relevant  Auditor  General  of  Alberta  recommendations  pertaining  to 
performance  measures  and  public  performance  reports  in  both  business  plans 
and  annual  reports,  and  indicate  the  status  of  their  implementation.  (Credibility) 


A Recommendation:  The  Government  of  Alberta  assume  a leadership  role  in  the 
development  of  performance  measures  for  horizontal  initiatives  (i.e.,  initiatives  in 
which  partners  from  two  or  more  organizations  agree  to  work  towards  the 
achievement  of  shared  outcomes).  (Credibility) 
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B Recommendation:  The  Ministry  of  Treasury  Board,  working  with  Executive 
Council,  assist  ministries  in  defining  and  articulating  the  ministry’s  strategic 
goals  and  performance  measures  so  that  users  of  public  performance  reports  can 
clearly  understand  how  a ministry’s  strategies,  programs  and  policies  contribute 
to  achieving  its  goals.  (Credibility) 


C Recommendation:  The  Government  of  Alberta  develop,  update  regularly,  and 
make  publicly  available  guidance  to  ministries  on  the  principles  and  best  practices 
for  effective  public  performance  reporting  to  the  legislature.17  (Credibility) 

MEDIUM-TERM 


E Recommendation:  The  performance  measures  used  in  PPRs  include  five  years  of 
historical  information  to  show  trends  and  ensure  consistency  for  year-to-year 
performance  measures.  (Credibility) 

F Recommendation:  In  addition  to  the  high-level  longer-term  outcome  measures, 
PPRs  include  more  performance  measures  where  some  change  can  be  effected 
within  the  3-year  business  planning  cycle.  (Credibility) 

N Recommendation:  Make  greater  use  of  hyperlinks  in  online  versions  of  PPRs  so 
users  can  “drill  down”  for  more  detailed  or  contextual  information.  (Clarity  and 
Accessibility) 


D Recommendation:  The  Ministry  of  Treasury  Board  review  ministry  business 
plans  and  annual  reports  to  determine  their  level  of  compliance  with  Treasury 
Board  guidelines  regarding  the  principles  and  best  practices  of  effective  public 
performance  reporting.  In  doing  so,  the  Ministry  of  Treasury  Board  still  ensures 
that  ministries  maintain  ownership  of  their  PPRs.  (Credibility) 

J Recommendation:  The  Government  of  Alberta  review  the  technical  background 
information  currently  made  available  to  assist  users  in  better  understanding  the 
information  in  PPRs.  (Relevance) 


U Recommendation:  Ministries  hold  information  sessions  for  stakeholder  user 
groups  when  PPRs  are  tabled  in  order  to  discuss  the  results  of  the  reports. 
(Engaging  Users) 

R Recommendation:  Undertake  a pilot  project  approach  to  public  performance 
reporting  consultations.  (Engaging  Users) 

V Recommendation:  Ministries  make  a concerted  effort  to  engage  MLA’s  in 
dialogue  on  goals,  expected  outcomes,  strategies,  performance  measures  and 
targets.  (Engaging  Users) 


T Recommendation:  The  Government  of  Alberta  identify  “champions”  within 

government  to  successfully  implement  new  approaches  to  PPRs.  (Engaging  Users) 

H Recommendation:  When  preparing  its  annual  report,  the  Government  of  Alberta 
compare  planned  versus  actual  spending,  list  what  goals,  strategies  and 
performance  measures  targets  were  met  or  not  met,  and  explain  why.  (Credibility) 


17  See  the  Appendix  for  a summary  of  the  Government  of  Canada  Treasury  Board  Secretariat’s  six  principles  for  effective  public 
performance  reporting. 
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LONGER-TERM 


G Recommendation:  The  Government  of  Alberta  link  financial  information,  such  as 
budgets,  more  closely  to  non-financial  information,  such  as  strategies, 
performance  measures  and  targets.  (Credibility) 

M Recommendation:  The  Government  of  Alberta  include  more  program-level 
information  and  data  on  the  internet,  and  link  it  to  the  goals,  strategies  and 
performance  measures  found  in  the  business  plans  and  annual  reports.  (Clarity 
and  Accessibility) 

Q Recommendation:  Future  stakeholder  consultations  related  to  new  policies  and 
programs  include  an  explicit  discussion  of  performance  measures,  targets  and 
reporting  mechanisms  and  how  frequently  performance  measures  and  targets 
should  be  reviewed.  (Engaging  Users) 


S Recommendation:  Common  principles  - such  as  a requirement  to  consult  a 

representative  group  of  stakeholders  - applied  across  ministries.  (Engaging  Users) 

L Recommendation:  The  Government  of  Alberta  improve  the  linkage  between 
business  plans  and  annual  reports  by  identifying  in  annual  reports  the 
assumptions  that  were  embodied  in  the  goals  and,  given  the  results,  what  actions 
would  be  taken  going  forward  to  achieve  the  goals.  (Relevance) 

W Recommendation:  Policy  Field  Committees  be  the  primary  forum  to  review 
stakeholder  opinions  about  performance  measures,  targets,  and  reporting 
mechanisms  as  part  of  their  mandate  to  review  annual  reports.  (Engaging  Users) 


FINDINGS  FALLING  BEYOND  THE  SCOPE  OF  THE  REPORT 

The  desire  for  more  program-level  statistics  and  information  was  an  issue  raised  repeatedly 
by  participants.  While  such  data  and  information  might  inform  the  development  of  PPRs,  it 
is  primarily  used  for  management  practices.  As  this  type  of  data  is  not  currently  available 
from  other  sources,  participants  have  tried  to  seek  it  through  PPRs.  This  influenced  the 
participants’  perceptions  of  PPRs.  As  such,  a specific  recommendation  related  to  this 
broader  need  is  to: 

♦ Examine  how  to  make  more  operational  information  available,  such  as  program  metrics. 
Technological  tools  may  be  available  to  help  make  operational  information  available 
outside  of  government,  subject  to  privacy  and  safety  considerations.  A cost-benefit 
analysis  should  be  undertaken  if  the  implementation  of  this  recommendation  is  likely  to 
prove  very  costly. 
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Appendices 

Consultation  approach  and  Methodology 

Intent 

For  the  Government  of  Alberta  (GoA),  accountability  is  a priority.  The  tools  and  practices 
that  enable  a government  to  be  accountable  to  its  citizens  evolve  over  time.  As  such,  it  is 
important  for  governments  to  review  their  accountability  tools  and  practices  to  see  in  what 
ways  they  may  be  enhanced  or  modernized. 

The  Government  of  Alberta’s  Treasury  Board  approached  CCAF  to  undertake  a direct 
dialogue  with  users  of  Public  Performance  Reports  (PPR).  Treasury  Board  and  CCAF  were 
natural  partners,  given  their  longstanding  collaboration  and  their  reputations  as  a leader  in 
this  field.  The  project  was  also  a fit  with  CCAF’s  Program  for  Improved  Public  Performance 
Reporting  under  the  auspices  of  a Sloan  Foundation  grant.  The  CCAF’s  performance 
reporting  website  www.performancereporting.ca  contains  additional  information  on  this 
broader  initiative. 

The  Ministry  of  Treasury  Board,  via  a working  group  and  an  advisory  committee,  led  and 
managed  all  aspects  of  the  consultation,  while  CCAF  and  Bluesky  Strategy  Group,  Inc. 
(Bluesky)  facilitated  at  the  consultations  and  interviews  and  prepared  this  report.  A series  of 
four  two-hour  consultation  sessions  were  held  in  Edmonton  and  Calgary  during  the  week  of 
October  26,  2007.  Each  session  had  four  to  eight  participants  divided  into  user  cohorts:  the 
media,  business  NGOs/professional  associations,  as  well  as  two  consultations  with 
social/advocacy  NGOs,  one  in  Edmonton  and  the  other  in  Calgary.  During  the  week  of 
December  10,  2007  two  more  consultations  with  user  cohort  groups  were  held,  one 
involving  legislators  from  opposition  parties  and  the  other  with  government  MLAs.  These 
consultations  were  complemented  by  several  individual  interviews  with  Cabinet  Ministers,  a 
senior  government  official,  and  Advisory  Committee  members. 

The  consultations  provided  an  opportunity  to  hear  directly  from  users  representing  a cross- 
section  of  interests  on  what  works,  what  does  not,  and  why.  This  was  one  of  the  first 
consultation  processes  of  its  kind  in  Canada. 

Currently,  the  Government  of  Alberta  prepares  a variety  of  public  performance  reports 
annually  (e.g.  business  plans,  budgets,  estimates,  and  annual  reports)  as  part  of  its 
commitment  to  be  accountable  and  in  keeping  with  certain  statutory  requirements. 

Research  conducted  previously  by  CCAF  and  others  indicates  that  legislators,  the  media 
and  the  public  generally  make  little  use  of  PPRs. 

The  goal  of  these  consultations  was  to  research,  report,  and  provide  recommendations  on 
improving  public  performance  reporting  in  the  Alberta  public  sector.  Specifically,  the  project 
evaluates  current  public  performance  reports;  identifies  what  aspects  of  current  reporting 
are  effective;  and  determines  what  improvements  can  be  made  to  meet  the  needs  of 
legislators,  the  media  and  the  public. 
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Assumptions 

The  scope  of  this  project  is  largely  limited  to  external  users  of  PPRs.  PPR  producers  and 
senior  decision-makers  in  government  were  not  widely  consulted.  They  are,  nonetheless,  a 
key  group  for  improving  PPRs.  Their  support  will  be  essential  if  the  benefits  of  this  project 
are  to  be  fully  realized.  They  will  need  to  be  willing  to  take  what  has  been  learned  in  these 
consultations  and  in  other  GoA  initiatives  to  enhance  PPRs  and  apply  it  to  Alberta’s  public 
performance  reporting  regime.  Without  a commitment  to  make  improvements,  the  value  of 
this  work  will  be  minimized. 

Much  effort  was  made  to  include  a representative  sample  of  users,  but  these  consultations 
were  more  qualitative  than  quantitative  - they  reflected  participants’  experiences  and 
knowledge  of  PPRs. 

Methodology 

The  consultations  were  moderated  by  Geoff  Dubrow,  Director  of  Capacity  Development  at 
CCAF,  and  by  Nicolas  Todd,  Senior  Consultant  at  Bluesky  Strategy  Group,  Inc.  At  each 
session  there  were  one  or  two  note-takers  to  ensure  the  participants’  remarks  and 
interactions  were  adequately  captured.  The  first  set  of  consultations  was  held  in  Edmonton 
and  Calgary  the  week  of  October  26,  2007.  Each  session  had  four  to  ten  participants,  each 
drawn  from  different  communities  of  practice.  The  groups  were: 

1 . Calgary-based  social/advocacy  non-governmental  organizations: 

2.  Edmonton-based  social/advocacy  non-governmental  organizations; 

3.  Members  of  the  media; 

4.  Edmonton-based  business  NGOs  and  professional  associations; 

At  the  four,  two-hour  consultations,  participants  responded  to  a written  questionnaire  that 
established  baseline  performance  measures  and  participated  in  a lively  dialogue.  They 
were  asked: 

♦ Do  you  use  performance  reports  and  for  what  purposes  do  you  use  them? 

♦ What  information  do  you  want  to  see  in  government  performance  reports? 

♦ What  information  do  you  consider  the  most  and  the  least  important  in  assessing  the 
performance  results  of  the  Alberta  government’s  policies,  programs  or  services? 

4-  Reviewing  some  of  Alberta’s  existing  PPRs,  what  type  of  information  from  these  reports 
do  you  find  useful  or  not  useful? 

♦ What  specific  content  or  processes  would  increase  or  decrease  your  confidence  in  the 
information  presented  in  these  documents? 

4-  What  format  or  layout  (e.g.,  graphs,  tables,  bullets,  text)  would  be  the  most  useful/least 
useful  when  receiving  this  information? 

4-  What  form  of  distribution  (e.g.,  e-mail,  internet,  and  hard  copy)  would  be  most 
useful/least  useful? 
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The  second  set  of  consultations  was  held  in  Edmonton  the  week  of  December  10,  2007. 

These  included: 

1 . Alberta  MLAs  belonging  to  parties  in  the  opposition; 

2.  Alberta  MLAs  from  the  governing  party,  and 

3.  Individual  interviews  with  two  Cabinet  Ministers,  a senior  government  official,  and 
Advisory  Committee  members. 

These  consultations  lasted  between  45  and  90  minutes,  during  which  time  participants 

were  asked: 

♦ Do  you  or  your  constituents  use  public  performance  reports?  If  so,  how?  How  relevant 
do  you  consider  the  reports  to  be  in  your  work? 

♦ What  information  would  help  you  better  assess  the  performance  results  of  the  Alberta 
Government’s  policies,  programs  or  services? 

♦ What  specific  content  or  processes  would  increase  your  confidence  in  these  reports? 

♦ What  role  should  the  legislature  have  in  the  creation  or  review  of  these  reports?  In 
performance  measures  or  targets  for  the  reports? 

♦ What  information  could  be  provided  in  these  reports  that  would  help  you  in  you  work 
with  constituents? 

♦ Can  the  legislature  play  a role  in  facilitating  public  input  into  these  reports? 

♦ What  tools,  format,  layout  or  method  for  distribution  would  be  best  for  you? 

The  participants  (other  than  the  Interviewees)  were  asked  to  complete  a Consultation 

Survey  prepared  by  the  Statistics  staff  at  Alberta  Finance.  The  Consultation  Survey  and  a 

sample  of  the  survey  are  attached  as  an  appendix  to  this  report. 
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Committee  Membership 

Advisory  Committee  Membership 

♦ Doug  Lynkowski,  Controller,  Treasury  Board  (Advisory  Committee  Chair) 

♦ Mark  Lisac,  Publisher  and  Editor,  Insight  into  Government 

♦ John  Watson,  Provincial  Audit  Committee 

♦ Tracey  Ball,  Provincial  Audit  Committee 

♦ Geoff  Dubrow,  Director,  Capacity  Development,  CCAF 

♦ John  Meston,  Executive  Director,  Alberta  Association  of  Services  for  Children  and 
Families 

Working  Group  Membership 

4 Murray  Lyle,  Performance  Planning  and  Reporting,  Alberta  Treasury  Board 
4 Tania  Arruda,  Performance  Planning  and  Reporting,  Alberta  Treasury  Board 
4 Anne  Bruce,  Performance  Planning  and  Reporting,  Alberta  Treasury  Board 
4 Bill  Hyshka,  Statistics,  Alberta  Finance 
4 Jolene  Court,  Statistics,  Alberta  Finance 
4 Sophie  Gaudet,  Statistics,  Alberta  Finance 
4 Kim  Lakeman,  Policy  Coordination,  Executive  Council  Office 
4 Geoff  Dubrow,  Capacity  Development,  CCAF 
4 Nicolas  Todd,  Bluesky  Strategy  Consulting,  Inc. 

Terms  of  Reference  for  the  Advisory  Committee 

Public  Performance  Reporting  Alberta  Project  Structure 
Advisory  Committee 

Responsibilities  - Advises  on  overall  project  direction  and  reviews  and  approves:  the 
questionnaire  used  for  the  consultations  with  legislators  and  members  of  the  media  and 
the  public  (NGOs);  the  membership  of  the  consultation  groups;  and  advises  on  the  final 
project  recommendations  to  be  implemented  and  the  content  of  the  Alberta  Final 
Report. 

Membership  - the  Advisory  Committee  will  consist  of  one  member  of  the  media;  audit 
committee  members;  a CCAF  representative;  an  individual  representing  NGOs;  and  a 
senior  government  official. 

Consultation  Groups 

Responsibilities  - The  three  Consultation  Groups  (legislators,  media  and  public)  will 
meet  at  least  once  for  about  two  hours.  Each  group  will,  through  a facilitator,  respond  to 
a set  of  survey  questions  and  participate  in  a group  discussion  regarding  public 
performance  reporting.  Prior  to  the  group  meeting,  each  member  will  receive  briefing 
materials  to  establish  common  reference  points. 

Membership  - one  Consultation  Group  will  consist  of  members  of  the  media,  another 
will  consist  of  members  of  the  legislature,  and  a third  will  consist  of  members  from  the 
public  (represented  by  a variety  of  NGOs  with  a session  in  Calgary  and  one  in 
Edmonton). 
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Working  Group 
Responsibilities  - 

4 Alberta  Treasury  Board  to  convene  and  form  the  nucleus  of  the  Working  Group; 

♦ Invite  individuals  to  participate  on  the  Advisory  Committee,  and  establish  the 
Committee; 

♦ Make  recommendations  to  the  Advisory  Committee  as  to  the  makeup  of  the 
legislative,  media  and  public  (NGOs)  consultation  groups; 

♦ Invite  attendees  to  participate  in  the  Consultation  Groups; 

♦ Develop  a suite  of  interview  questions  to  assist  in  focusing  discussions  of  the 
legislative,  media  and  public  (NGOs)  consultation  groups; 

♦ Develop  questions  for  the  written  Consultation  Survey; 

♦ Organize  the  Consultation  Groups; 

♦ Produce  a report  on  the  development  and  current  status  of  public  performance 
reporting  in  Alberta; 

♦ Gather,  analyze  and  produce  a report  on  the  Alberta  Project  findings; 

♦ Work  with  the  Advisory  Committee  to  review  recommendations  of  the  Alberta  Final 
Report;  and 

♦ Publish  the  project  findings. 

Membership  - It  is  proposed  that  the  Working  Group  consist  of  Performance  Planning 
and  Reporting,  Alberta  Treasury  Board;  Alberta  Finance,  Statistics;  Director  of  Capacity 
Development,  CCAF;  and  a representative  from  Executive  Council;  and  other  interested 
Alberta  government  officials. 

CCAF 

Responsibilities  - 

♦ Provide  advice  on  the  Advisory  Committee  membership; 

♦ Participate  on  the  Advisory  Committee; 

♦ Provide  advice  in  developing  Consultation  Group  questionnaires; 

♦ Provide  advice  on  the  makeup  of  the  legislative,  media  and  public  (NGOs) 
Consultation  Groups; 

4-  Facilitate  the  Consultation  Group,  including  note-taking  and  summarize  the  findings; 
and 

4-  Prepare  final  report. 

Membership  - CCAF  officials  and  consultants. 

A Word  of  Thanks  to  the  Advisory  Committee 

CCAF  and  the  Government  of  Alberta  would  like  to  thank  the  members  of  the  Advisory 
Committee  for  their  ongoing  advice  and  insights,  which  have  proved  invaluable.  Several 
members  were  able  to  provide  their  insights  and  observations  on  PPRs  through  informal 
interviews. 
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Public  Performance  Reporting  Consultation  Survey  Results 

Introduction 

This  report  highlights  the  survey  results  of  the  Public  Performance  Reporting,  Consultation 
Survey;  conducted  by  Alberta  Finance,  Statistics  on  behalf  of  Alberta  Treasury  Board. 

There  were  a series  of  six,  two-hour  focus  group  consultation  sessions  with  each  session 
consisting  of  the  following  user  cohorts: 

♦ The  media  - print,  TV,  radio,  etc.  (7  participants) 

♦ Non-government  organizations  (NGOs)  - one  session  with  business/professional  NGOs 
(8  participants),  and  two  social  policy  NGOs  one  in  Edmonton  and  one  in  Calgary 

(10  participants) 

♦ Alberta  MLAs  - opposition  members  (3  participants) 

♦ Alberta  MLAs  - government  members  (3  participants) 

The  survey  was  administered  at  the  end  of  each  focus  group  consultation  session.  A copy 
of  this  survey  can  be  found  in  Appendix  III. 

Survey  Responses 

Respondents  were  asked  to  complete  the  survey  at  the  end  of  their  consultation  session.  A 
total  of  31  surveys  were  completed  giving  a 100%  response  rate.  The  consultation 
respondent  sample  was  not  designed  to  be  statistically  representative  of  each  group. 

Results  by  Section 

The  survey  consisted  of  nine  sections  including  a section  for  additional  comments/ 
suggestions  and  respondent  profile.  Section  topics  included  knowledge,  use,  format, 
confidence  in  the  Public  Performance  Reports  (PPRs).  The  following  provides  highlights 
and  a detailed  analysis  of  each  section. 

Section  1:  Are  Alberta’s  PPRs  being  read? 

The  PPRs  read  by  the  respondents  prior  to  the  consultation  are  listed  in  order  of  the  PPR 
most  frequently  read.  The  percent  frequency  of  each  PPR  read  is  shown  in  parentheses. 

1.  Ministry  Business  Plan(s)  (87.1%) 

2.  Ministry  Annual  Report(s)  (83.9%) 

3.  Government  of  Alberta  Strategic  Business  Plan  (74.2%) 

4.  Government  of  Alberta  Measuring  Up  Report  (71.0%) 

The  most  frequently  read  report,  prior  to  the  consultation,  was  the  Ministry  Business 
Plan(s)  followed  closely  by  Ministry  Annual  Report(s). 

Overall,  of  the  PPRs  read  prior  to  the  consultation,  none  were  below  70%  readership  and 
only  6.5%  of  the  respondents  read  “None  of  the  Above”  PPRs  (see  Figure  1).  These  results 
indicate  an  overall,  relatively  strong  readership  for  the  PPRs. 
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The  two  reports  that  had  statistically  significant  differences  between  the  groups  were;  the 
Government  of  Alberta  Strategic  Business  Plan  and  Ministry  Business  Plan(s).  The 
significant  difference  noted  was  the  media  group  reported  a higher  percent  having  not  read 
these  two  PPRs,  when  compared  to  the  other  groups. 


Figure  1:  Percent  of  participants  that  read  PPRs 


GOA  Strategic  GOA  Measuring  Up  Ministry  Business  Ministry  Annual  None  of  Above 

Business  Plan  Report  Plan(s)  Report(s) 


Public  Performance  Reports 


Section  2:  The  extent  to  which  PPRs  have  met  user  needs 

Respondents  were  asked  to  rate  their  level  of  agreement  that  the  reports  they  read  have 
met  their  needs.  The  ratings  were  converted  into  an  agreement  index  that  ranged  from 
0 to  100,  where  0 was  low  agreement  and  100  was  high  agreement.  The  following  are  the 
PPRs  in  order  of  high  to  low  agreement,  beginning  with  the  highest  mean  (average)  score. 

1.  Ministry  Business  Plan(s)  (54.3) 

2.  Government  of  Alberta  Strategic  Business  Plan  (50.4) 

3.  Ministry  Annual  Report(s)  (47.7) 

4.  Government  of  Alberta  Measuring  Up  Report  (43.8) 

Overall,  respondents  were  divided  between  “Somewhat  Agree”  and  “Somewhat  Disagree”, 
with  mean  (average)  scores  between  54.3  and  43.8  (see  Figure  2A). 

Figure  2A:  Percent  of  participants  reporting  PPRs  have  met  their  needs 
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The  Government  of  Alberta  Measuring  Up  Report  met  respondent  needs  the  least, 
displaying  the  lowest  mean  (average)  score  of  43.8.  It  also  had  the  highest  level  of 
disagreement  selected  by  respondents  (see  Figure  2B). 

As  indicated  in  Figures  2A  and  2B,  it  was  agreed  most  of  the  time  by  respondents,  that  the 
reports  they  read  have  met  their  needs. 


There  were  no  statistically  significant  differences  noted  in  responses  between  the  groups 
for  Section  2 questions. 


Figure  2B:  Have  PPRs  met  user  needs? 
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Section  3:  What  PPRs  are  currently  used  for 

Respondents  were  asked  to  rate  how  frequently  they  have  used  any  of  the  PPRs  for 
various  purposes.  The  ratings  were  converted  into  a frequency  index  that  ranged  from 
0 to  100,  where  0 was  low  frequency  and  100  was  high  frequency. 

The  following  are  the  level  of  frequency  for  the  defined  “purpose”  categories,  from  highest 
to  lowest,  beginning  with  highest  mean  (average)  score. 

1.  Information  purposes  (65.5) 

2.  Identifying  priorities  (49.1) 

3.  Policy  development  (46.3) 

4.  Performance  assessment  (37.5) 

5.  Development  of  programs  & services  (35.0) 

6.  Resource  allocation  decisions  (21.9) 

Overall,  the  most  frequent  purpose  for  the  use  of  the  reports  was  for  “information 
purposes”,  which  had  highest  mean  (average)  score  of  65.5.  It  also  had  the  highest  number 
of  respondents  chose  “Always”  (see  Figure  3). 

The  least  frequent  purpose  for  the  use  of  the  reports,  which  had  the  lowest  mean  (average) 
score  of  21.9  and  highest  selection  of  “Never”  and  “Seldom”,  was  for  “resource  allocation 
decisions”  (see  Figure  3). 
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Overall,  survey  responses  indicated  that  most  respondents  are  using  the  PPRs  for 
“information  purposes”  and  are  rarely  using  them  for  “resource  allocation  decisions”. 

Some  of  “Other”  purposes  of  use  specified  by  respondents  were  (in  no  specific  order): 

♦ "Identifying  gaps  of  service." 

♦ "To  learn  about  the  department." 

♦ "Information  not  decision  making." 

The  three  reports  that  had  statistically  significant  differences  between  the  groups  surveyed 
were;  “policy  development”,  “identifying  priorities”,  and  “resource  allocation  decisions”. 

A significant  difference  was  noted  for  the  media  which  “Never”  used  the  reports  for  the 
three  purposes.  In  addition  to  the  media,  NGOs  also  “Never”  and  “Seldom”  used  PPRs  for 
“resource  allocation  decisions”. 

These  statistical  results  could  generally  be  expected  as  both  the  media  and  NGOs  are 
generally  not  involved  in  “policy  development”,  “identifying  priorities”,  and  “resource 
allocation  decisions”  decisions. 


Figure  3:  How  frequently  have  you  used  PPRs  for. . . 


Policy  Development 
Development  of  Programs/Services 
Information  Purposes 
Identifying  Priorities 
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Section  4:  Opportunities  to  increase  the  use  of  Alberta’s  PPRs 

Respondents  were  asked  to  rank,  in  order  of  importance  (1 , 2,  3...),  what  would  increase 
their  use  of  Alberta’s  PPRs.  The  rankings  were  converted  into  an  “importance”  index  that 
ranged  from  0 to  100,  where  0 was  least  important  and  100  was  most  important. 

The  most  important  ranked  factor  which  would  increase  the  use  of  Alberta’s  PPRs  was  if 
“results  are  presented  in  a straightforward  and  simple  manner”.  This  was  also  ranked 
highest  as  “Most  Important”  and  “Important”  (see  Figure  4). 

The  least  important  ranked  factor  for  increasing  use  of  Alberta’s  PPRs  is  if  “results  include 
assessments  of  all  policy,  programs  and  service  areas”.  This  also  had  the  highest  ranking 
of  “Least  Important”  (see  Figure  4). 


IMPROVING  PUBLIC  PERFORMANCE  REPORTS 


The  rankings  generally  indicate  that  most  respondents  primarily  want  “results  that  are 
presented  in  a straightforward  and  simple  manner”  in  order  to  increase  their  use  of 
Alberta’s  PPRs. 

Some  of  the  “Other”  increase  uses  specified  by  respondents  were  (in  no  specific  order): 

♦ "Clearer  identification  of  budget  allocations  to  programs." 

♦ "Benchmarking  to  other  provinces." 

♦ "Right  things  are  being  measured." 

4-  "Results  - performance  measurements  having  a meaningful  relationship  to  results  trying 
to  be  achieved.  Uniform  reporting  of  annual  reports." 

-4  "Measures  better  reflect  government's  accountabilities  as  opposed  to  system 
outcomes." 

4-  "More  relevant  data." 

4-  "Consistent  from  year  to  year;  measures  directly  related  to  department  function;  better 
explanation/justification  at  why  measure  chosen;  some  3rd  party  assessment." 

4-  "Relevance  is  it  consequential  to  my  audience?" 

4-  "Results  show  past  targets  and  whether  they've  been  met." 

4-  "Level  of  independence/3rd  party  verification." 

4-  "Outcome  measures." 

4-  "Results  are  relevant." 

4-  "Do  not  change  every  year." 

There  were  no  statistically  significant  differences  in  responses  between  the  groups  for 
Section  4 questions. 


Figure  4:  What  would  increase  use  of  Alberta’s  PPRs?  If  results. . . 
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Section  5:  What  format  would  you  like  Alberta’s  PPRs  in? 

Respondents  were  asked  to  rank,  in  order  of  preference  (1 , 2,  3...),  what  format  would  they 
prefer  Alberta’s  PPRs.  The  question  was  broken  down  into  three  areas:  Content, 
Presentation  and  Distribution. 
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Content  - Respondents  ranked  the  most  preferred  content  for  Alberta’s  PPRs  as  “report 
card/aggregated  indices  report”.  The  content  ranked  as  least  preferred  was  equally  divided 
between  “detailed/descriptive  report”  and  “technical/data  oriented  report”.  However, 
“technical/data  oriented  report”  had  more  respondents  rank  it  as  “Most  Important”,  than  for 
“detailed/descriptive  report”. 

Figure  5A:  Format  respondents  would  like  Alberta’s  PPRs  in  (content) 
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Presentation  - The  presentation  format  that  respondents  ranked  they  would  most  prefer  to 
see  Alberta’s  PPRs  in  is  “graphs”.  Format  ranked  least  preferred  is  in  “text”. 


Distribution  - The  most  preferred  method  of  distribution  was  “internet”.  The  least  ranked 
was  “hard  copy”. 

Figure  5C:  Format  respondents  would  like  Alberta’s  PPRs  in  (distribution) 
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Some  of  the  “Other”  format  suggestions  specified  by  respondents  for  all  three  sections 
were  (in  no  specific  order): 

♦ “Blend  of  descriptive  technical/data.” 

♦ “Graphical/scorecard  representations.” 

4 “Combination  of  above.” 

4 “Depends  on  the  information  - different  information  suited  to  different  displays.” 

4-  “What  is  the  trend,  where  are  we  going?” 

4-  “Email  notification  of  availability.” 

4 “Email  link  to  full  report  online.” 

4 “CD.” 

Overall,  the  rankings  appear  to  indicate  most  respondents  primarily  prefer  the  format  to  be 
a “report  card/aggregated  indices  report”  with  “graphs”  available  on  the  Internet.  There 
were  no  statistically  significant  differences  in  responses  between  the  groups  for  ail  three 
areas  for  Section  5 questions. 

Section  6:  Effect  on  use  if  Alberta’s  PPRs  are  changed 

Overall,  most  respondents  stated  that  they  would  use  Alberta’s  PPRs,  if  changes  were 
made,  “Occasionally  (2-3  times  a year)”.  None  of  the  respondents  stated  that  they  would 
“Never”  use  Alberta’s  PPRs,  if  changes  were  made. 


Never  Seldom  Occasionally  Sometimes  Frequently 


The  “Comments”  provided  by  respondents  were  (in  no  specific  order): 

4 "The  mandate  letters  are  prime  source  of  strategic  planning  process  and  then  how  do 
they  appear  in  ministry  business  plan." 

4 "Use  mainly  to  align  with  internal  strategic  plan  once  a year.  Although  quarterly 
summaries  would  be  useful  (one  pager)." 

4 "This  won't  change  my  use  - do  see  value  - enhancing  creditability  of  measures." 

4 "Response  and  utilization  depends  largely  upon  the  selection  of  indicators  and  their 
relevance  to  the  intervention  and  effectiveness  of  government  policy.” 

4 "Depending  on  the  current  activities/mandates  of  ministry  to  my  organization." 

4 "4  -5  times  a year  is  an  approximation  - would  keep  them  handy  as  a continued 
reference." 
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♦ "I'm  highly  skeptical  that  the  changes  will  be  made." 

*4  "Would  have  to  see  them  first." 

There  were  no  statistically  significant  differences  in  responses  between  the  groups  for 
Section  6. 

Section  7:  What  would  increase  your  confidence  in  Alberta’s  PPRs? 

Respondents  were  asked  to  give  two  suggestions  that  would  increase  their  confidence  in 
Alberta’s  PPRs.  Responses  from  respondents  were  placed  into  one  of  the  seven 
categories  listed  below.  Percentage  of  responses  for  each  category  is  in  parentheses. 

4 Independent  Review/Input  (21.3%) 

4-  Relevant  Measures/Indicators  (19.1%) 

4-  Consistency  (14.9%) 

4-  Sourcing  (Detailed/Additional)  (12.8%) 

4 Accountability  (10.6%) 

4-  Simplified  Wording  (6.4%) 

4-  “Other”  (14.9%) 

When  asked  what  would  increase  their  confidence  the  most,  the  largest  number  of 
responses  for  any  given  category  was  “independent  review/input”  followed  closely  by 
“relevant  measures/indicators”  (see  Figure  7). 

Overall,  based  on  an  analysis  of  the  total  responses  given,  respondents  cited  they  would 
generally  be  more  confident  in  Alberta’s  PPRs  if  there  was  “independent  review/input”  and 
if  it  contained  “relevant  measures/indicators”. 

Samples  of  “Other”  suggestions,  not  categorized,  were  (in  no  specific  order): 

4-  “Acknowledgment  of  limitations”. 

4-  “Targets  set  collaboratively  before”. 

4-  “Evaluate  qualitatively  and  quantitatively”. 

4 “Criteria  for  performance  measures”. 

4-  “Public  disclosure”. 

There  were  no  statistically  significant  differences  in  responses  between  the  groups  for 
Section  7. 


IMPROVING  PUBLIC  PERFORMANCE  REPORTS 


Accountablity  Independent  Relevant  Sourcing  Consistency  Simplified 

Review/  Input  Measures/  Wording 

Indicators 


Comments/Suggestions 

The  following  are  additional  comments/suggestions  provided  by  some  of  the  respondents 

for  what  would  make  Alberta’s  PPRs  more  user-friendly  (in  no  specific  order). 

♦ “Plain  language  is  essential.  Simplify  the  verbiage.  Report  on  outcomes  not  process. 
Make  reporting  interactive  and  web-based  - new  media.  Consider  getting  rid  of  one-off 
annual  reporting  and  replace  with  year-round  progress  updates  a few  measures  at  a 
time.  Include  feedback  mechanism.” 

♦ “Community/NGO  consultation  on  measures  and  outcomes.  Non  government  contractor 
creating  the  report.  Both  qualitative  and  quantitative  measurable.  Measurable/objectives 
that  are  more  specific.  Comparisons  with  best  practices  throughout  Canada.  Written  in 
plain  language.  Explanation  as  to  when  outcomes  aren't  being  met  (why)  and  actual 
reporting  if  they  aren't  being  met.  Don't  report  on  process  to  make  it  sound  as  a 
success,  or  measurable.  Shorter  report  that  is  printed  in  a more  economical  format  (not 
high  gloss  with  aesthetics  being  first  concern).  Quarterly  reporting  and/or  consultation 
via  web  so  there  is  no  printing  cost.  Presently  no  mention  on  the  collaborative 
relationships  and  NGO's  that  helped  to  achieve  these  goals  (and  most  are  not 
government  funded  to  do  that).  Programs  and  innovations  should  be  reported  on  review 
reports  from  other  jurisdictions.” 

♦ “Use  the  Internet  only  if  it  would  allow  both  users  who  want  "snapshots"  and  those  who 
require  detailed  research  and  explanation  of  where  and  how  data  is  sourced.  Very 
interesting  to  see  if  the  2007/08  PPR's  reflect  the  changes  provided  by  new  provincial 
leadership  - openness,  transparency,  etc.” 

♦ “More  web-usage  for  wider  availability.  Cut  back  on  wordiness.  Executive  summaries. 
Combine  reports  where  possible  to  eliminate  all  of  the  duplication  amongst  all  four 
reports.” 

♦ “Ensure  access  to  supporting  data.  Hyperlink  or  electronic  version.  More  attempts  to 
present  objective  report  and  less  as  a P.R.  exercise.” 

♦ “The  criteria  and  performance  measures  need  to  be  set  in  consultation  with  the  public 
and  key  stakeholders  who  are  affected  by  the  policies.  Once  these  performance 
measures  are  set  they  need  to  be  produced  by  the  departments  in  ways  that  shows  how 
we  are  doing  in  each  area.  If  we  are  not  succeeding  in  each  area  then  be  willing  to 
admit  this.  The  report  should  encourage  innovation  not  limit  the  departments  to  a narrow 
list  of  limited  goals.  This  must  be  removed  from  any  control  of  the  Public  Affairs  Bureau.” 
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4 “Seems  obvious  many  around  the  table  did  not  use/see  these  documents  before  - 
would  like  to  see  better  job  of  communicating  on  the  measures.  How  and  why  they  are 
chosen.  How  and  why  targets  are  set.  How  departments  are  held  accountable  for 
outcomes.  Small  testing  on  measurements  - might  be  useful  - building  confidence. 

More  thought  on  how  to  determine  what  outcomes  government  should  strive  for  when 
their  role  is  as  facilitators  - should  measures  of  programs  delivered  be  considered  as 
opposed  to  the  aggregate  outcomes?” 

4 “Indicators  we'd  like  to  see  Ell/Advanced  Education/Education.  Data  sources  that  shows 
what  skills  (occupations,  employers  are  trying  to  fill  and  how  well  these  three 
departments  are  doing  with  matching  immigrants/graduates  to  need).  Education  - 
number  of  students  in  career  and  technology  studies.  Measures  that  show  success  at 
hiring  from  underrepresented  groups  (including  immigrants  - disabilities)  and  targeting 
programs  for  them.” 

4 “No  additional  comments,  but  I appreciated  the  opportunity  for  input.” 

4 “In  my  section  the  most  critical  issue  is  that  government  reports  of  indicators  over  which 
it  has  little  direct  influence.  The  output  of  the  ministry  is  policy  and  finance  - the 
effectiveness  of  those  policies  and  financial  decisions  need  to  be  measured  more 
directly.  Much  basic  information  about  education,  particularly  teaching  and  learning 
conditions,  is  not  collected  (see  vacant  data  fields  in  the  OECD  report  on  education).” 

4 “I  would  like  to  see  greater  consistency  surrounding  the  format  and  distribution  for  the 
annual  reports.  They  can  be  found  on-line,  but  are  often  in  various  formats  making 
ministry  to  ministry  comparisons  challenging.” 

4 “I  really  appreciate  being  involved  in  this  process.  Thank  you.” 

4 “In  general,  they  should  seem  less  designed  for  political  purposes  and  more  a genuine 
management  tool.  Creditability  is  the  major  factor  in  user  friendliness.” 

4 “I  like  the  idea  of  standard  releases  instead  in  all  at  once.  I feel  I'm  at  a disadvantage  to 
media  organizations  that  have  a large  staff.  If  I don't  have  the  story  on  day  one,  then  I’m 
likely  to  disregard  every  thing  after  that.... "that's  not  a story".” 

4 "Target  your  audience  with  appropriate  information.” 
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Public  Performance  Reporting 
Consultation  Survey 


To  assist  the  Government  of  Alberta  in  improving  the 
usability  of  the  Public  Performance  Reports  (PPRs), 
please  provide  your  views  on  the  following  aspects. 


This  survey  is  being  conducted  as  part  of  the  consultation  on  strengthening  Alberta's  Public  Performance 
Reports.  Please  do  not  put  your  name  on  this  survey.  The  information  collected  through  this 
consultation  will  be  presented  in  aggregate  form  in  the  final  report.  Comments  or  opinions  collected  at 
the  consultations  and  presented  in  the  final  report  will  not  be  attributed  to  any  individual  or  organization. 


ydlbsrra 

TREASURY  BOARD 
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Have  you  read  any  of  the  following  Public  Performance  Reports  (PPRs)  prior  to  this  consultation? 

Government  of  Alberta  Strategic  Business  Plan 
Government  of  Alberta  Measuring  Up  Report 
Ministry  Business  Plan(s) 

Ministry  Annual  Report(s) 

None  of  the  above 


If  you  have  used  any  of  the  above  PPRs,  on  the  scale  provided  please  indicate  to  what  extent  you  agree 
they  have  met  your  needs. 


1 

Strongly 

2 

3 

Somewhat 

4 

Somewhat 

5 

6 

Strongly 

Disagree 

Disagree 

Disagree 

Agree 

Agree 

Agree 

The  following  PPRs  have  met  my  needs:  1 2 3 4 5 6 N/A 

a)  The  Government  of  Alberta  Strategic  Business  Plan 

b)  The  Government  of  Alberta  Measuring  Up  Report 

c)  A Ministry  Business  Plan(s) 

d)  A Ministry  Annual  Report(s) 


\ 

If  you  have  used  any  of  the  above  PPRs,  please  indicate  how  frequently  they  have  been  used  for  the 

following  purposes. 

1 2 3 

4 

5 

Never  Seldom  Sometimes 

Usually 

Always 

1 have  used  a PPR  for: 

1 2 

3 4 5 N/A 

a) 

policy  development 

b) 

development  of  programs  and  services 

c) 

information  purposes  (media,  public  consultations, 
personal  informative  purposes) 

d) 

identifying  priorities 

e) 

resource  allocation  decisions 

0 

performance  assessment 

9) 

other:  (please  specify) 

V 

J 

Please  rank  (1,  2,  3...)  in  order  of  importance  what  would  increase  your  use  of  Alberta’s  PPRs. 

results  can  be  compared  against  other  published  results 

results  include  assessments  of  all  policy,  program,  and  service  areas 

results  are  current 

results  are  presented  in  a straightforward  and  simple  manner 

other:  (please  specify) 
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For  each  of  the  following  sections,  please  rank  in  order  of  preference  (1,  2,  3...)  in  what  format  you 
would  like  the  Alberta's  PPRs: 

a)  Content: 

Report  Card  / Aggregated  Indices  Report 

Detailed  / Descriptive  Report 

Technical  / Data  Oriented  Report 

Other:  (please  specify) 

b)  Presentation: 

Numbers 

Text 

Graphs 

Other:  (please  specify) 

c)  Distribution: 

E-mail 

Internet 

Hard  copy 

Other:  (please  specify) 


If  the  changes  you  have  identified  above  were  made  to  Alberta's  PPRs,  how  often  would  you  use  them? 

Seldom  Occasionally  Sometimes  Frequently 

Never  (Once  a year)  (2-3  times  a year)  (4-5  times  a year)  (6  or  more  times  a year) 

Comments: 


List  2 things  that  would  increase  your  confidence  in  Alberta's  PPRs: 

\ 

a) 

b)  

/ 

r 

\ 

Please  indicate  which  group  you  represent: 

Government  Representative 

Media  / Communications 

Non-Government  Organization 

Other 

V 

/ 
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Please  provide  any  additional  comments  or  suggestions  that  you  feel 
would  be  useful  in  making  Alberta's  PPRs  more  user-friendly. 
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Existing  Research  findings  and  Existing  practices 

Retrospective  on  the  GoA’s  Public  Performance  Reports 

In  the  early  1990s,  the  Government  of  Alberta  identified  a need  for  better  tracking  of  and 
reporting  on  results  of  government  programs  and  services.  In  1993,  the  government 
created  the  Financial  Review  Commission,  which  recommended  actions  needed  to  improve 
the  province’s  management  and  reporting  systems.  Budget  roundtables  were  held  in  1993 
and  again  in  1994  with  the  media,  academics,  business  and  community  leaders,  and  senior 
government  officials.  Based  on  those  consultations,  the  government  took  some  significant 
steps  toward  improving  its  public  accountability.  It  introduced  the  Government 
Accountability  Act,  developed  the  first  set  of  three-year  ministry  business  plans,  and 
released  Measuring  Up,  Alberta’s  first  annual  public  performance  report  and  the  first  of  its 
kind  in  Canada. 

The  objective  of  introducing  the  legislation  was  to  require  the  government  to  produce 
documents  on  a regular  basis  that  disclosed  to  the  public,  through  the  Legislative 
Assembly,  the  planned  and  actual  results  of  government  policies,  programs  and  services, 
thereby  improving  their  effectiveness  in  meeting  the  needs  of  Albertans.  Performance 
measurement  was  instituted  as  part  of  the  larger  business  planning  process,  with  the 
intention  of  making  government  more  open  and  accountable.  The  first  annual  Measuring 
Up  report  (Government  of  Alberta,  1995)  contained  22  core  government  performance 
measures  under  18  goals.  Measuring  Up  contained  a mix  of  economic,  environmental  and 
social  indicators,  intended  to  provide  a snapshot  of  overall  government  performance. 

The  priority  between  1993  and  2001  was  on  developing  integrated  and  strategic  business 
plans,  with  a focus  on  outcomes,  targets  and  performance  measures.  The  intention  of 
moving  to  a performance-based  accountability  framework  was  to  provide  timely  and  factual 
financial  and  non-financial  information  about  the  government’s  use  of  public  funds  and 
stewardship  of  the  province’s  public  resources.  Another  intention  was  to  start  Albertans 
thinking  in  terms  of  the  results  provided  by  government  as  opposed  to  just  spending. 
Performance  measurement  was  central  to  the  accountability  framework  so  that  the 
government’s  performance  could  be  measured,  publicly  reported  and  evaluated. 

Challenges  during  this  time  included:  developing  a few  relevant  performance  measures  to 
use  in  a public  report  to  reflect  the  government’s  key  accomplishments,  and  disclosing  what 
was  accomplished  when  there  is  much  about  government  that  cannot  be  measured  directly 
or  quantitatively. 

Between  2002  and  2006,  the  government  of  Alberta  focused  on  assessing,  maintaining, 
growing  and  sustaining  Alberta’s  accountability  and  fiscal  framework.  Business  plan 
standards  and  a reference  guide  were  developed  to  provide  a framework  to  develop 
consistency  (in  quality,  look  and  form)  and  mandated  components  in  all  ministry  business 
plans.  As  a result,  plans  could  more  easily  be  understood  by  the  public  and  there  was  a 
shift  from  more  “operational”  plans  to  more  strategic  business  plans. 

Integration  of  financial  and  non-financial  results  is  viewed  as  strengthening  performance 
reporting,  so  by  the  2004-07  Government  of  Alberta  Strategic  Business  Plan,  each  of  the 
goals  was  aligned  with  expenditures  in  the  Fiscal  Plan  linking  the  financial  information  in 
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the  fiscal  plan  with  non-financial  performance  information  in  the  Business  Plan  (using 
Statistics  Canada’s  Expense  by  Function  criteria).  Ministries  also  began  to  integrate 
financial  and  non-financial  results  by  providing  expense  by  core  business  breakdowns  in 
their  business  plans  and  annual  reports. 

An  unresolved  challenge  during  this  time,  which  continues  to  be  a challenge  today,  was  in 
regard  to  attribution  of  results  to  government  policies,  programs  and  outcomes,  as  this  can 
be  ambiguous.  Going  forward,  a number  of  ongoing  challenges  remain  including: 

1 . The  challenge  of  making  these  reports  more  relevant  by  expanding  and  improving  the 
performance  story  through  the  use  of  other  sources  of  information  (e.g.,  evaluation 
results,  external  studies,  etc.)  while  still  keeping  the  report  readable. 

2.  A challenge  pertaining  to  the  credibility  of  these  reports,  that  is,  the  performance 
information  presented  must  be  perceived  to  be  free  from  bias.  The  public  continues  to 
be  unconvinced  that  the  public  performance  reports  are  accountability  documents. 

They  view  them  as  communications  documents,  despite  review  by  the  Office  of  the 
Auditor  General. 

3.  The  increasing  need  to  distinguish  between  performance  budgeting  and  management, 
which  are  decision-making  functions,  and  public  performance  reporting  for 
accountability.  The  performance  information  used  for  decision-making  is,  in  many  cases, 
different  from  what  is  required  for  accountability. 

Existing  Research  Findings 

CCAF  research  indicated  that  there  are  several  reasons  or  barriers  impeding  legislators’ 
use  of  public  performance  reports  and,  in  general,  it  often  is  due  to  a combination  of 
legislator’s  incentives,  capabilities,  interests,  and  time  constraints. 

Some  of  the  barriers  identified  in  CCAF’s  research  as  impeding  legislators’  use  of  public 
performance  documents  included: 

1 . Reports  are  not  well  aligned  with  how  legislators  think  and  should  be  presented  in  more 
verbal,  anecdotal,  and  concrete  terms. 

2.  With  the  current  size,  format,  and  number  of  reports,  there  is  not  enough  time  for 
legislators  to  read  and  assimilate  the  information  they  receive. 

3.  Legislators  feel  that  reports  lack  credibility  and  are  a spin  product  of  a ministry’s  public 
relations. 

4.  There  are  few  rewards  or  incentives  for  legislators  who  scrutinize  reports. 

5.  There  is  insufficient  staff  to  support  the  research  and  analysis  of  reports,  which  is  often 
critical  in  helping  legislators  gain  a comprehensive  understanding  of  the  material. 

CCAF  research  found  that  there  are  several  reasons  or  barriers  that  impede  the  media  and 
the  public  from  using  and  understanding  public  performance  reports. 

1 . Journalists  have  commented  that  reports  are  too  abstract  and  lack  information  media 
find  interesting.  This  includes  failing  to  clearly  demonstrate  program  outcomes  or 
results,  lack  of  rationale  for  government  objectives,  as  well  as  inadequate  reasons  for 
why  the  government  has  undertaken  certain  activities. 

2.  Media  lack  confidence  in  government  performance  reports  that  are  based  on 
evaluations  performed  internally.  Journalists  may  use  them  for  statistical  information,  but 
would  not  likely  depend  on  them  for  substance. 
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3.  Due  to  a shift  from  investigative/analytical  journalism  to  covering  just  the  immediate 
story,  the  needed  expertise  and  allocation  of  time  is  not  currently  sufficient  in  order  to 
look  in-depth  at  government  business. 

4.  Reports  generally  do  not  cover  issues  that  concern  the  general  public. 

5.  Government  outcomes  are  often  stated  but  are  not  necessarily  proven  or  directly 
attributable  to  the  related  policy  or  program. 

6.  Public  performance  results  may  acknowledge  shortcomings,  but  solutions,  or  the 
solicitation  of  solutions  are  not  usually  offered. 

With  information  gathered  from  the  media  and  the  public  and  related  sources,  with 
identification  of  the  extent  to  which  these  barriers  exist  in  Alberta,  and  with  a willingness  to 
make  changes  based  on  these  findings,  Alberta’s  public  performance  reports  can  be 
improved  to  overcome  the  barriers. 

Existing  Practices 

Many  participants  in  the  consultations  anticipated  that  an  all-party  committee  (i.e.,  the 
Policy  Field  Committees)  could  play  a role  in  public  performance  reporting.  In  this  context, 
legislative  and  executive  branch  officials  in  Alberta  may  want  to  examine  the  roles  of  other 
legislative  bodies  in  Canada.  Two  notable  examples  - the  British  Columbia  Select  Standing 
Committee  on  Crown  Corporations  and  the  House  of  Commons  Standing  Committee  on 
Government  Operations  and  Estimates  - are  highlighted  here. 

British  Columbia  Select  Standing  Committee  on  Crown  Corporations  - The  British 
Columbia  Select  Standing  Committee  on  Crown  Corporations  was  appointed  to  review  the 
annual  reports  and  performance  plans  of  British  Columbia  Crown  Corporations.  The 
Committee  can  call  Ministers,  senior  ministry  officials,  and/or  Crown  Corporation  executives 
to  appear  as  witnesses.  The  Committee  has  the  mandate  to  review  Commercial  and 
Service  Delivery  Crown  Corporation  Service  Plans  and  Annual  Reports. 

House  of  Commons  Standing  Committee  on  Government  Operations  and 
Estimates  - The  House  of  Commons  Standing  Committee  on  Procedure  and  House 
Affairs  conducted  an  in-depth  study  on  the  business  of  supply  entitled  The  Business  of 
Supply:  Completing  the  Circle  of  Control,  in  December  1998. 18  The  report  provides  a history 
of  parliament’s  work  on  supply  and  estimates  and  includes  recommendations  on 
strengthening  parliament’s  role.  It  recommended  that  parliament  locate  “within  a single 
committee,  broad  responsibilities  relating  to  the  supply  process,  and  financial  reporting  to 
Parliament  by  government  organizations.”19 

The  House  of  Commons  Standing  Committee  on  Government  Operations  and  Estimates 
tabled  an  insightful  report  in  September  2003.20  Entitled  Meaningful  Scrutiny:  Practical 
Improvements  to  the  Estimates  Process,  it  makes  numerous  recommendations,  some  of 
which  are  summarized  below: 


18  Available  online  at:  http://www.parl.gc.ca/lnfoComDoc/36/1/PRHA/Studies/Reports/prharp51-e.htm. 

19  The  mandate  of  the  Standing  Committee  on  Government  Operations  and  Estimates  is  at:  http://cmte.parl.gc.ca/cmte/ 
CommitteeList.aspx?Lang=1&PARLSES=391&JNT=G&SELID=e20_&COM=10468 

20  The  Report  is  available  at:  http://cmte.parl.gc.ca/cmte/CommitteePublication.aspx?COM=3278&Lang=1&Sourceld=213502 
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♦ That  the  document  entitled  Parliamentary  Committee  Review  of  the  Estimates 
Documents  - a guide  from  the  Office  of  the  Auditor  General  of  Canada  to  assist 
Members  of  Parliament  (MPs)  in  becoming  more  involved  in  the  estimates  review  - be 
provided  by  the  Auditor  General  to  all  parliamentarians,  after  each  election,  as  a 
reference  tool. 

4-  That  the  House  of  Commons  and  Library  of  Parliament  collaborate  with  Treasury  Board 
Secretariat  to  include  a session  on  the  estimates  process  in  the  orientation  provided  to 
newly  elected  Members  of  Parliament,  and  that  follow-up  training  focussed  on  practical 
approaches  to  maximizing  Parliament’s  effectiveness  in  holding  governments 
accountable  through  the  estimates  process  be  provided  at  regular  intervals  each  year, 
funded  by  a reallocation  from  the  budget  of  the  Canadian  Centre  for  Management 
Development  (or  its  successor). 

♦ That  the  parliamentary  committees  reviewing  the  Estimates  of  large  departments 
consider  limiting  their  study  to  one  program  or  one  agency  in  particular  (selected  in 
compliance  with  the  principles  of  alternation  and  sampling),  in  light  of  the  timeframe  and 
resources  available. 

4-  That  parliamentary  committees  consider  holding  a planning  meeting  before  the  hearing 
with  public  servants  that  would  enable  them  to  learn  more  about  the  program  or  agency 
to  be  reviewed.  Such  planning  meetings  could  use  documents  obtained  beforehand 
from  the  officials  in  answer  to  specific  questions. 

4-  That  the  members  of  parliamentary  committees  consider  the  possibility  of  dividing  up 
the  tasks  involved  in  the  budget  review  and  that  they  do  the  same  with  the  additional 
documents  provided  by  the  departments  or  agencies. 

4-  That  parliamentary  committees  consider  the  possibility  of  asking  researchers  to  collect 
information  on  the  program  or  agency  under  review  and  to  draft  technical  and 
administrative  questions  to  be  forwarded  by  the  committee  chair  to  departmental 
officials  before  they  appear  before  the  Committee. 

4-  That  parliamentary  committees  consider  preparing  a precise  meeting  schedule  for  the 
Estimates  review  in  their  work  plan  and  that  the  public  servants  and  experts  called  to 
appear  be  informed  as  far  in  advance  as  possible. 

4-  That  parliamentary  committees  consider  tabling  short  reports  on  departmental  plans 
and  priorities  and  performance  reports  as  a routine  practice,  in  order  to  provide 
departments  with  clear  feedback  on  their  central  accountability  documents. 

Principles  for  Effective  Public  Performance  Reporting 

Public  Sector  Accounting  Board’s  Public  Performance  Reporting  Assessment 

Guide 

The  PSAB  Guide21  outlines  13  key  evaluation  questions: 

1 . Is  the  report  easily  accessible  and  identifiable  as  the  entity’s  Annual  Performance 
Report? 

2.  Does  the  report  provide  information  that  appears  reliable  and  valid? 

3.  Is  the  entity’s  performance  information  relevant? 

4.  Does  the  entity  provide  fair  information  in  its  performance  report? 

5.  Is  the  entity’s  performance  information  comparable  and  consistent? 


21  The  Guide  is  available  at:  http://www.psab-ccsp.ca/download.cfm?cijd=37690&lajd=1&re_id=0 
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6.  Is  the  report  understandable? 

7.  Does  the  report  focus  on  the  few  critical  aspects  of  performance? 

8.  Does  the  report  describe  the  entity’s  strategic  direction? 

9.  Does  the  entity  explain  actual  results  for  the  reporting  period  and  compare  them  with 
planned  results,  explaining  any  significant  variances? 

10.  Does  the  report  provide  comparative  information  about  trends,  benchmarks,  baseline 
data,  or  the  performance  of  other  similar  organizations? 

1 1 . Does  the  report  describe  lessons  learned  and  key  factors  influencing  performance  and 
results? 

12.  Did  the  entity  link  its  financial  and  non-financial  performance  information? 

13.  Was  the  basis  for  reporting  disclosed? 

Government  of  Canada’s  Six  Principles  for  Effective  Public  Performance 

Reporting 

The  Government  of  Canada’s  Treasury  Board  Secretariat  has  adopted  six  principles  for 

effective  public  performance  reporting22  to  parliament.  They  are: 

1 . Benefits/Results  for  Canadians:  describe  the  activities  and  outputs  that  the  organization 
intends  to  undertake  or  produce  over  the  planning  period.  The  information  must  help 
Canadians  understand  what  the  department  (and  hence  government)  seeks  to  achieve 
by  providing  a high  level  summary  of  the  planned  results  being  sought  and  the 
strategies  and  resources  to  be  used. 

2.  Plans  and  priorities  that  will  be  used  to  deliver  the  Benefits/Results  to  Canadians: 
Outline  a plan  to  deliver  results  to  Canadians.  Highlight  the  results  to  be  achieved  by 
providing  a clear  target.  In  all  cases  an  estimated  timeframe  should  be  established  for 
the  completion  of  the  plan  and  some  indication  should  be  made  as  to  when  Canadians 
can  expect  to  see  results. 

3.  Lessons  learned  and  applied  from  past  experiences:  Explain  the  context  for  those 
choices.  Revising  and  improving  plans  is  an  indication  of  sound  management  practices. 
Departmental  plans  are  expected  to  change  in  response  to  a changing  environment  or 
adapting  to  lessons  learned  and  those  changes  should  be  described.  It  is  important  to 
identify  the  changes  and  explain  the  rationale  for  them. 

4.  Challenges,  risks  and  the  rationale  for  the  choices  made:  A strong  plan  notes  the 
various  options  and  highlights  the  logic  behind  the  choice  made. 

5.  Total  planned  spending:  Illustrate  the  logical  link  between  what  the  department  intends 
to  accomplish  and  the  resources  available  in  support  of  its  strategic  outcomes.  Identify 
the  resources  allotted  to  each  departmental  priority  within  each  strategic  outcome.  Only 
main  or  key  priorities  need  to  be  listed,  but  provide  an  idea  of  how  much  a particular 
priority  is  expected  to  cost. 

6.  Assessing  performance:  Explain  how  the  organization  intends  to  monitor  its  progress 
toward  achieving  the  results  set  out.  Appropriate  monitoring  relies  on  a well-reasoned 
set  of  indicators  that  measure  actual  outcomes. 


22  The  2007/8  Guide  is  available  at:  http://www.tbs-sct.gc.ca/rpp/0708/guide/guide_e.asp 
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Government  of  Alberta’s  Criteria  for  Selecting  Public  Performance  Measures 

The  GoA’s  Measuring  Performance  - A Reference  Guide23  is  a “reference  guide  on 
measuring  government  performance  in  Alberta”  and  “discusses  the  role  of  performance 
measures  within  the  context  of  the  government's  business  planning  process,  and  how 
performance  measures  can  facilitate  and  enhance  program  delivery  and  management.” 

The  following  is  an  extract  of  what  the  Guide  says  about  developing  and  selecting 
performance  measures. 

Guiding  Principles 

♦ Focus  on  results  - determine  the  effects  programs  are  having  rather  than  measuring 
what  has  been  produced. 

♦ A few  key  measures  per  ministry  - provide  a snapshot  of  the  ministry's  performance  for 
its  core  businesses. 

♦ Measures  developed  by  ministry  program  officials  - programs  officials  know  their 
business  the  best. 

4 Measures  owned  by  ministries  - ministries  held  accountable  for  the  measures  they 
develop  and  the  results  they  achieve. 

4 Measures  should  be  free  from  bias  - report  both  good  and  bad  performance. 

4 Work  with  the  Auditor  General  - to  ensure  the  selection  of  valid  and  objective  measures. 

Key  Criteria  for  the  Selection  of  Measures: 

4 Understandability  - the  measure  and  information  are  clear  and  easily  understood  by  the 
public,  and  sufficiently  explain  how  performance  is  being  assessed. 

4 Relevance  - the  measure  is  an  accurate  representation  of  what  is  being  measured.  The 
information  presented  is  timely  and  directly  related  to  the  subject  matter. 

4 Reliability  - the  information  is  free  from  error,  unbiased  and  complete.  Also,  the  results 
can  be  duplicated  by  others  using  the  same  information  and  methodology. 

4 Comparability  - results  can  be  compared  to  other  years  or  to  similar  organizations. 
Questions  to  Help  Select  Performance  Measures: 

4 Do  the  measures  relate  to  the  stated  core  businesses  and  goals? 

4 Does  the  measure  make  sense  and  is  the  wording  understandable? 

4 Does  the  measure  really  indicate  the  effects  government  intends  the  program  to  have? 
4 Is  the  outcome  measured  at  least  partially  within  the  organization's  ability  to  influence? 

4 Can  the  measure  show  the  extent  to  which  goals  have  been  achieved? 

4 Is  the  data  accurate  and  can  the  information  be  collected  over  time  on  a consistent 
basis? 

4 Has  the  data  been  impartially  gathered  and  analyzed? 

4 Will  the  measures  be  valid  for  more  than  one  period  without  significant  changes? 


23  The  GoA’s  Guide,  published  in  1997,  is  available  at:  http://www.finance.alberta.ca/publications/measuring/measupgu/ 
pfmguide.pdf 
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♦ Do  the  measures  allow  for  comparisons  with  past  performance,  other  organizations, 
other  jurisdictions? 

♦ Can  others  using  the  same  data  arrive  at  similar  results  or  conclusions? 

♦ Is  the  cost  of  collecting  the  information  reasonable? 

♦ Do  the  measures  provide  performance  information  on  ministry/government  priorities? 

♦ Will  the  information  be  available  on  a timely  basis? 


IMPROVING  PUBLIC  PERFORMANCE  REPORTS 


59 


List  of  Consultation  Participants 

Media 

♦ CHED  Radio 

♦ Edmonton  Journal 

♦ CKUA 

♦ Alberta  Political  Scan 

♦ Insight  Into  Government 

♦ CBC  Radio 

♦ CTV 

Business  NGOs/Professional  Associations 

♦ Canadian  Taxpayers  Federation  (Alberta) 

4-  Alberta  Teachers  Association 

4-  Canadian  Federation  of  Independent  Business 
4-  Alberta  Association  of  Municipal  Districts  and  Councils 

♦ Certified  General  Accountants  of  Alberta 

4-  Canadian  Home  Builders  Association  (Alberta) 

4-  Alberta  Union  of  Provincial  Employees 
4-  Retail  Council  of  Canada  (Alberta) 

Social/advocacy  NGO 

4-  Public  Interest  Alberta 
4-  Alberta  Council  on  Aging 

♦ Catholic  Social  Services 
4-  Muttart  Foundation 

4-  Volunteer  Alberta 

4“  Alberta  Cancer  Board  and  Foundation 
4-  Calgary  Chamber  of  Voluntary  Organizations 
4-  Prospect  Human  Services  Society 
4-  United  Way  of  Calgary  and  Area 
4-  Volunteer  Calgary 

Members  of  the  Legislative  Assembly  of  Alberta 

4-  Hon.  Carol  Haley,  MLA  for  Airdrie-Chestermere 
4-  Hon.  Gordon  Graydon,  MLA  for  Prairie-Wapiti 
4-  Hon.  Victor  Doerksen,  MLA  for  Red  Deer-South 
4-  Hon.  Mo  Elsalhy,  MLA  for  Edmonton-McClung 
4-  Hon.  Dan  Backs,  MLA  for  Edmonton-Manning 
4-  Hon.  Hugh  Macdonald,  MLA  for  Edmonton-Gold  Bar 

Individual  Interviews 

4-  Hon.  Ron  Stevens,  Deputy  Premier  and  Minister  of  Justice  and  Attorney  General,  MLA  for 
Calgary-Glenmore 

4-  Hon.  Lloyd  Snelgrove,  President  of  Treasury  Board,  Minister  of  Service  Alberta,  Minister 
Responsible  for  Corporate  Human  Resources,  MLA  for  Vermilion-Lloydminster 
4-  Mr.  Ron  Hicks,  Deputy  Minister  of  Executive  Council 

4-  Ms.  Tracey  Ball,  Member  of  the  Provincial  Audit  Committee,  Member  of  the  Public 
Performance  Reporting  Consultation  Advisory  Committee 
4-  Mr.  John  Watson,  Member  of  the  Provincial  Audit  Committee,  Member  of  the  Public 
Performance  Reporting  Consultation  Advisory  Committee 
4-  Mr.  John  Meston,  Executive  Director,  Alberta  Association  of  Services  for  Children  and 
Families,  Member  of  the  Public  Performance  Reporting  Consultation  Advisory  Committee 
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